On the Eﬀectiveness of Accuracy ofAutomated Feature Location Technique
Takashi Ishio∗ , Shinpei Hayashi† , Hiroshi Kazato‡ , Tsuyoshi Oshima§∗ Osaka University, Osaka 565–0871, Japan

Email: ishio@ist.osaka-u.ac.jp
Email: hayashi@se.cs.titech.ac.jp‡ NTT DATA INTELLILINK CORPORATION, Tokyo 104–0052, Japan Email: kazatoh@intellilink.co.jp§ NTT Software Innovation Center, Tokyo 180–8585, Japan Email: oshima.tsuyoshi@lab.ntt.co.jp† Tokyo Institute of Technology, Tokyo 152–8552, Japan

Abstract—Automated feature location techniques have been
proposed to extract program elements that are likely to be
relevant to a given feature. A more accurate result is expected
to enable developers to perform more accurate feature location.
However, several experiments assessing traceability recovery haveshown that analysts cannot utilize an accurate traceability matrix
for their tasks. Because feature location deals with a certain typeof traceability links, it is an important question whether the same
phenomena are visible in feature location or not. To answer thatquestion, we have conducted a controlled experiment. We haveasked 20 subjects to locate features using lists of methods ofwhich the accuracy is controlled artificially. The result diﬀers
from the traceability recovery experiments. Subjects given an
accurate list would be able to locate a feature more accurately.
However, subjects could not locate the complete implementationof features in 83% of tasks. Results show that the accuracy of
automated feature location techniques is eﬀective, but it might
be insuﬃcient for perfect feature location.
Index Terms—feature location, impact analysis, program comprehension, human factor

I. Introduction
Feature location is a program understanding phase in software maintenance. A feature represents a functionality that isdefined by requirements and which is accessible to developers
and users [1]. Locating the current implementation of a feature
in source code is important for developers to perform various
maintenance tasks such as enhancement, bug fixing, andrefactoring related to the feature. Mäder et al. reported thatdevelopers who know source files related to requirements can
produce a software change more eﬃciently [2]. Therefore,feature location in this paper denotes a phase to find codesnippets relevant to a feature to the greatest extent possible
before source code modification.
Although feature location is important, locating the complete implementation of a feature is diﬃcult for developers [3],[4]. According to Wang et al. [3], given a feature, developers
use a keyword search tool to identify seed methods that arelikely to be relevant to the feature. For each seed method,developers explore its source code and related methods tovalidate whether the methods are actually relevant to thefeature or not. Although each step in the process seems simple,
developers must identify relevant methods before they canascertain the complete implementation of a feature. Furthermore, industrial developers are often asked to locate features

978-1-4799-2931-3/13 c 2013 IEEEin an unfamiliar system using only its source code because anenterprise application might outlive its development team.
To support developers locating a feature, various automatedfeature location techniques have been proposed [1]. A typicaltechnique takes keywords or a description of a feature as
input and extracts a list of methods that are only relevantto the feature. For example, Marcus et al. [5] proposed
an application of latent semantic indexing (LSI) to feature
location. Poshyvanyk et al. [6] proposed the use of execution
traces of a target program to improve a ranking obtainedby LSI. Eaddy et al. [7] combined static analysis, dynamic
analysis, and an information retrieval (IR) technique to extract
a better ranking. Developers can investigate source code usinga result of these techniques, e.g. the top ten methods of aranking.
A more accurate result of automated feature location techniques is expected to enable developers to perform moreaccurate feature location. However, a question remains as towhether the accuracy of automated feature location techniques
actually contributes or not to the performance of developers.
This question arises from an observation reported by Cuddeback et al. [8], Kong et al. [9], and Dekhtyar et al. [10]. They
asked analysts to do manual validation of a traceability matrix
between requirements and system tests. The results indicatedthat, whereas analysts given a less-accurate traceability matrix
can identify many false positives and false negatives in thematrix, analysts given an accurate traceability matrix tend to
decrease the overall accuracy of the matrix. If the same phenomena are visible in feature location, then accurate automatedfeature location results might be less eﬀective for actual feature
location performed by developers.
To answer the question, we conducted an experiment with
20 subjects in three organizations. We asked the subjects tolocate a feature using a given list of methods by excluding
false positives from the list and by identifying false negatives
in source code. To evaluate the influence of accuracy of lists,
we prepared a pair of tasks that specify the same feature to
be located but which provide diﬀerent lists of methods. The
lists are derived artificially from the result of a textual searchusing LSI, so that the lists have diﬀerent accuracy, but havethe same length. The features and the result of LSI used inthis experiment are involved in the open dataset of the workby Dit et al. and Gethers et al. [1], [11], although we have

381

WCRE 2013, Koblenz, Germany

Accepted for publication by IEEE. c 2013 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

extended the descriptions of features. Each subject used either
an accurate list or a less-accurate list for a feature.
Consequently, we obtained the following observations.
• Subjects given an accurate list located a feature moreaccurately than subjects given a less-accurate list.
• Subjects missed one or more relevant methods in 83 of
100 tasks. Even if an accurate list covered the complete
implementation of a feature, several relevant methods
have been falsely recognized as false positives.
• Subjects improved precision of the final result by excluding false positives. However, they did not improve recall
in several features.
The result shows that developers can utilize results ofautomated feature location techniques. However, developers
tend to miss methods that are relevant to a feature if its
description is not suﬃciently clear for developers.
The contributions of this paper are summarized as follows.
• We have shown the manner in which accuracy of automated feature location techniques contribute to the accuracy of feature location tasks performed by developers.
• We have reported the manner in which we have preparedfeature location tasks from an existing benchmark. It isuseful for researchers to conduct a similar experimentusing the benchmark.
• We have made our dataset freely available so that otherresearchers can replicate the experiment1 .
The sections are organized as follows. Section II describes
related work and our research background. We state ourresearch questions in Section III. Section IV explains thesetting of our experiment. Section V shows the results ofthe experiment. In Sections VI and VII, we discuss theobservations and the threats to the validity of our experiment.
Section VIII describes conclusions and future work.
II. Related WorkIn software maintenance, developers must understand unfamiliar source code of a target system. Ko et al. [12]reported that developers often use a keyword search tool toidentify source code relevant to their tasks. Wang et al. [3]emphasized the process of manual feature location in software
maintenance. They also observed that developers often used a
keyword search tool to identify seed methods that are likely
to be relevant to features.
Several researchers reported that developers were unable
to identify the complete implementation of features. Wang etal. [3] observed that each of precision and recall is less than
75% in their feature location tasks. Egyed et al. [4] conducted
an experiment of manual recovery of requirements-to-code
links. Compared with the correct links created by developersof the programs, subjects recognized 95% of irrelevant pairs ofrequirements and classes, whereas the subjects missed about
half of the relevant pairs. Lindvall et al. [13] conducted acase study of manual impact analysis. The result shows that
1 http://sel.ist.osaka-u.ac.jp/FL/

developers were able to predict only a half of classes to bemodified for the next maintenance release.
Various automated feature location techniques have been
proposed in the literature [1]. Some are comparable in precision and recall to manual feature location. SNIAFL [14]shows that its precision and recall are, respectively, 91% and
99% for several small programs. CERBERUS [7] shows thatits precision and recall are, respectively, 75% and 73% in thebest configuration for a particular set of concerns. Gethers etal. [11] compared several impact analysis techniques in morepractical settings. The best analysis automatically identified
41–75% of methods modified for a feature request. Revelle etal. [15] manually evaluated results of several automated feature location techniques. They reported that the best technique
identified three relevant methods among the top ten methods.
Many researchers investigated whether developers can utilize a result of automated techniques or not. Revelle et al. [15]reported that the authors’ and several students’ validation
results agreed over 90% of the time for a certain feature.
The observation is promising but not generalizable because
it is a single case and the authors have identified only true
positives in the results rather than the complete implementationof features. In traceability research, Cuddeback et al. [8]investigated manual validation of the requirement traceability
matrix representing links between requirements and systemtests. They reported that analysts given an accurate traceability
matrix decreased the overall accuracy of the matrix. Kong etal. [9] analyzed the process of the traceability validation tasks.
Dekhtyar et al. [10] confirmed the observation by statisticalanalysis. The analysis indicates that developers cannot utilize
accurate traceability links. Based on the results describedabove, Cuddeback et al. [16] discussed a means of addressing
the inaccuracy of developers. Ghabi et al. [17] proposed
automated validation for traceability recovery.
Parnin et al. [18] conducted a controlled experiment toevaluate the usefulness of automated debugging techniques.
The experiment used an artificially modified ranking derived
from a result of an automated technique. It showed that aranking change did not aﬀect the performance of developers.
Chatterji et al. [19] reported that developers were not able touse code clone detection for bug fixing with no training in code
clones. These research eﬀorts also indicated the importance ofhuman study to evaluate the actual usefulness of automatedsupport for developers.
In our experiment, we used LSI to extract a list of methods
relevant to a feature. LSI-based feature location was proposedby Marcus et al. [5]. Poshyvanyk et al. [6] combined dynamic
analysis with LSI to improve ranking. Binkley et al. [20]showed that identifier normalization is eﬀective to improvesearch results. The eﬀectiveness of LSI was also confirmed
in fault localization. Beard et al. [21] reported that LSI canrecommend an appropriate starting point to locate a fault insource code for 60 out of 63 bugs in a system. To keep our
experiment simple and easy to replicate, we have used a simple
LSI approach that computes similarity among methods and thedescription of a feature [11].

382TABLE I
Features, Tasks, and Their Accuracy
SystemjEdit

muCommander

Feature (Issue ID)
f J0 (2122926)# methods (Type)1

f J1 (1747300)

13 (larger)

f J2 (2668434)

6 (smaller)

f J3 (1593464)
f M0 (311)102

f M1 (60)32 (larger)

f M2 (231)

6 (smaller)

Task (Type)J0 (example)
J1b (better)J1w (worse)
J2b (better)J2w (worse)
J3 (goldset)
M0 (example)
M1b (better)
M1w (worse)
M2b (better)M2w (worse)

Precision
0.33 (1/3)
1.00 (10/10)
0.40 (4/10)
0.60 (6/10)
0.30 (3/10)
1.00 (10/10)
0.33 (1/3)
1.00 (10/10)
0.80 (8/10)
0.60 (6/10)
0.30 (3/10)

Recall1.00 (1/1)
0.77 (10/13)
0.31 (4/13)
1.00 (6/6)
0.50 (3/6)
1.00 (10/10)
0.50 (1/2)
0.31 (10/32)
0.25 (8/32)
1.00 (6/6)
0.50 (3/6)

F-measure
0.50
0.87
0.35
0.75
0.38
1.00
0.40
0.48
0.38
0.75
0.38

IV. Experimental Setup

III. Research Questions
We have conducted a controlled experiment on feature
location using human subjects. We have defined a processof feature location supported by an automated technique asfollows: given an initial list of methods produced by an automated technique, a developer validates the list by removing
irrelevant methods (false positives) from the list and by adding
missing relevant methods (false negatives) to the list. Ideally,a validated list should include no irrelevant methods but all
methods relevant to a feature.
We formulated the following four research questions.
RQ1 Do better initial precision and recall engender better
performance in feature location by developers?
RQ2 Which option is more important for feature location,
initial precision or recall?
RQ3 How do developers spend time to validate a list of
methods?
RQ4 How does a validated list diﬀer from its initial list?
We carefully set up feature location tasks of several types;
each task is validation of an initial list of methods in a limited
time. To answer RQ1, we introduced two categories of tasks:
better and worse. The initial lists of better tasks are accurate,i.e., having better precision and recall values, than those ofworse tasks. We compared the F-measure of the validated lists
between those of better and worse tasks.
For a response to RQ2, we also introduced two categories
of tasks: precision-intensive and recall-intensive. The initiallists of precision-intensive tasks have high precision but low
recall; they are related to little false positives and many false
negatives. In contrast, the initial lists of recall-intensive tasks
have high recall but low precision. We compared F-measuresof the validated lists between precision-intensive tasks andrecall-intensive tasks.
To answer RQ3, we have analyzed the activities of subjects.
Each subject is asked to input their judgments in our Eclipse
plug-in. We have analyzed a series of judges with timestamps
to elucidate how they functioned and whether subjects had
suﬃcient time or not.
To answer RQ4, we have analyzed the manner in which
precision and recall are improved or degraded by validation
tasks.

A. Features
The features for our study were prepared using an existingchange-history-based feature location benchmark [1]. Thisbenchmark is constructed using the information of changes inrevision control systems such as Subversion, and the related
tickets in issue tracking systems such as Trac or Bugzilla. The
benchmark provides a list of change requests including feature
requests and bug reports. Each change request is associatedwith a list of methods modified to implement the change, ora goldset. For objectivity, we have used the existing dataset.
The seven extracted features are shown in the left columnsof Table I. The symbol of a feature is used to identify thefeature in the remaining part of the paper. The Issue ID of afeature indicates the identifier of the feature in the benchmark.
We have selected five features for the experiment from twosystems: jEdit and muCommander. We extracted a larger
feature and a smaller feature from each system to extract
precision-intensive and recall-intensive tasks. f J1 and f M1 arelarger features, of which goldsets have more than ten methods.
f J2 and f M2 are smaller features, of which goldsets have fewer
than ten methods. The feature f J3 is extracted for checking thecases in which subjects receive the true goldset having just ten
methods.
In addition to these five features, we have selected twotiny features: f J0 implemented in a single method and f M0
implemented in two methods. These features are used to
explain the process of feature location to subjects.
To use the benchmark for our experiment, we fixed two
problems in the benchmark as follows:

383•

Curation of goldsets. Some goldsets of the benchmark
include inappropriate methods. Because a goldset of afeature is generated automatically from a change set ofsource code in a revision history, it includes methods
that are irrelevant to the target feature if developers
commit multiple intentional changes at once, e.g. theimplementation of the feature together with a refactoring.
Additionally, if rename refactorings are performed after
implementing the feature, then some methods in thegoldset might no longer be found in the source code. We
have manually identified renamed methods and updated

muCommander Feature #231
Short Description“Skip all” for errors that occur during a file transfer operation
Long Description
As suggested in the [http://www.mucommander.com/forums/
viewtopic.php?f=2&t=938 forums]: adding a “Skip all” button
when an error occurs in a multiple file move / copy operation
would be a nice feature to have.
Feature Description
muCommander has a feature that copies/moves files selected by auser. When a user tries to execute a copy, muCommander shows a
dialog to specify a destination directory. Pushing the Copy buttonin the dialog starts a copy process. If an error occurred during thecopying of a file, then an error dialog shows a message and asks
the user to skip the file, retry to copy the file, or cancel the copy
process.
The new feature is “Skip All.” The dialog to specify a destination
directory has a new check box with the caption is “Skip errors.” If
a user checked the box, then muCommander automatically skips
a file if an error occurred, without showing a dialog. The error
message dialog also has a new button “Skip all.” If the button
is pushed, then muCommander shows no error dialog in further
errors, as “Skip errors” is checked.

10
Better task
false positives

Smaller feature(Goldset)size < 10

Worse task
false positives

Better task
Larger feature(Goldset)size > 10

Worse task
false positives

Fig. 2. Generation of better and worse tasks.

Letting g, i, and t be the size of a goldset, an initial list, andtrue positives, respectively, then the initial F-measure is givenby the harmonic mean of precision t/i and recall t/g:
F=

Fig. 1. Descriptions of feature f M2 .

such goldsets. It is noteworthy that the number of methods
in Table I reflects this manual refinement.
• Addition of extended descriptions. Some descriptionsin the benchmark do not describe the associated features
accurately. Because descriptions are submitted to an issue
tracking system by a requester of the feature, a gap separates the description and the actually implemented feature.
So that subjects can correctly understand features fromtheir descriptions, we provided an extended description
for each feature in addition to the original description.
An example of the extended description is presented
as “Feature Description” in Figure 1. For all extended
descriptions, we used a single paragraph to describe thebasic behavior without the associated feature and anotherparagraph to describe the target feature to be located.
The extended descriptions are based on the recordedchanges in the benchmark. We tried to explain all thefunctionalities added by the recorded changes although
the original descriptions do not mention some of them.
Each feature comprises the following elements: (1) a short
description in the benchmark, (2) a long description in thebenchmark, (3) an extended description, (4) a screenshot ofan execution, and (5) the goldset.
B. Tasks
From the extracted features tasks are generated. The input
for a task is a pair of a feature and an initial list of methods forthe feature. The goal of a task is to validate the initial list using
source code of a system and the descriptions of the feature.
To normalize the eﬀort of subjects, every task is associatedwith ten methods, except for example tasks.
We evaluate the performance of a task using precision,
recall, and F-measure for the initial list and the validated list.

2 · precision · recall
2t=.
precision + recall
g+i

By eliminating t using 2t = F · (g + i), precision and recall canbe written in terms of F, i, and g:g
F · (g + i) F 
1+ ,precision =
=
2i
2
i
i
F · (g + i) F
1+.
recall ==2g
2g
In the experiment, we controlled F as an independent variable
and fixed i at ten. Then the tradeoﬀ between precision andrecall is determined by g, where precision gets better for larger
features and recall does better for smaller ones.
We have generated better and worse tasks for each feature.
The lists in better tasks include methods relevant to the features(gold methods) to the greatest extent possible, whereas the lists
in worse tasks include several methods that are irrelevant to
the features (false positives). A notational example is shownin Figure 2 to illustrate how a pair of tasks is generated fromthe goldset of a feature. In the figure, a gray box and a whitebox respectively represent a gold method and a false positive.
For smaller features having fewer than ten gold methods,
generated better tasks have all the gold methods and some false
positives. In contrast, for larger features having more than ten
gold methods, generated better tasks have ten gold methods.
The unused gold methods are regarded as false negatives.
Generated worse tasks have some gold methods and false
positives. We have injected false positives into worse tasks
until their F-measures become less than the threshold FT . We
use FT = 0.40 to diﬀerentiate the number of false positives inbetter and worse tasks clearly.
The precision, recall, and F-measure of the initial lists ofthe generated tasks are shown in the right columns of Table I.
The tasks generated from larger features include a few false
positives but fail to capture many of gold methods. In contrast,
the tasks generated from smaller features cover almost all of
their goldsets, but include many false positives. Therefore, theformer and latter tasks are regarded respectively as precisionintensive and recall-intensive.

384

TABLE II
Task Assignment
Subject
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Organization

Osaka University

Tokyo Institute ofTechnology

NTT

1st
J1b
J2b
J2w
J1w
M2b
M1b
M1w
M2w
J2b
J1b
J1w
J2w
M1b
M2b
M2w
M1w
J1b
J2b
M1b
M2w

2nd
J2b
J1w
J1b
J2w
M1b
M2w
J2b
M1w
J1b
J2w
J2b
J1w
M2b
M1w
M1b
M2w
J2w
J1w
M2b
M1w

3rd
M1w
M1b
M2b
M1b
J1w
J2b
J2w
J2b
M2w
M1w
M1b
M1b
J2w
J2w
J1w
J2b
M2b
M2w
J1w
J1b

4th
M2w
M2w
M1w
M2b
J2w
J1w
J1b
J1b
M1w
M2b
M2w
M2b
J1w
J1b
J2b
J1b
M1w
M1b
J2w
J2b

5th
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3
J3

Fig. 3. Screenshot of FLPlayer.

We have used LSI to select gold methods and false positives
included in the initial lists. Instead of computing LSI result
by ourselves, we used the LSI result prepared by Gethers etal. [11], which is an extension of the benchmark for impact
analysis. In Gethers’s dataset, an LSI result for a feature
is a ranking of methods with contents similar to the longdescription of the feature. We have selected an appropriate
number of gold methods and false positives from the topof the ranking. For example, J1w includes a list of fourgold methods and six false positives. The gold methods havehigher similarity than the nine other relevant methods. The
false positives have higher similarity than other irrelevant
methods. The generated initial lists are sorted by their LSI
ranking. Almost of gold methods occur in lower rank thanfalse positives in initial lists because the LSI ranking of thegold methods are almost lower than ten.
C. Subjects and Task Assignment
We recruited 20 subjects from three organizations of bothacademia and industry. Subjects included 16 students ofsoftware engineering and 4 industrial developers. Their Java
experience was widely distributed from 2 to 16 years, with a
median of three years.
No subject knew the target systems. This situation is common in software maintenance tasks. Developers might have to
update legacy software developed by other teams.
Each subject examined the five assigned tasks in the orderpresented in Table II. We carefully assigned the tasks for eachsubject, satisfying the following constraints:
Every subject examines the first and second tasks of thesame system followed by the third and fourth tasks of theother system, and ends with the goldset task (J3 ).
• Every subject covers all of the five features.
• Every subject experiences all of the diﬀerent types oftasks: a better precision-intensive task, a worse precision•

intensive task, a better recall-intensive task, and a worse
recall-intensive task.
• Every task is examined by at least ten subjects.
D. Environment
Subjects are given an Eclipse IDE installed with a special
view named FLPlayer. We chose Eclipse as our environment
because of its publicity and familiarity to subjects. FLPlayer isa view for validating a list of methods in Eclipse. A screenshotof the view is presented in Figure 3. Subjects can see a listof methods as a table in the view. When they double-click amethod in the view, a source code editor automatically opens
and moves to the definition of the method. After investigating
the source code, they answer whether the method is relevant
or irrelevant to the feature, using the drop-down menu in thesecond column of the view. For example, in the figure, atleast six methods are enumerated. Two of them are specified
as irrelevant. In addition, subjects can add a method that is
not listed in the view from the context menu of the method ina source code editor.
For each task, every subject is given a printed document fora task and Eclipse environment in which FLPlayer is showing
the list of methods ordered by their LSI ranking. The document
included the descriptions of a feature, a screenshot, the same
list of methods as shown in FLPlayer, and a quick reference
guide of Eclipse and FLPlayer. The feature description foreach task are translated into Japanese due to the nationality ofthe subjects. The subjects are prohibited to access other online
materials such as an issue tracking system and to execute thetarget systems. We did not provide the LSI scores to subjects
and used them just for the order of methods because theymight reveal which methods are injected as false positives.
Eight laptop computers of the same model were used forthe experiment, each equipped with Core i5 processor (Intel
Corp.), 4 GB RAM, 256 GB SSD, and a 12.1-inch LCD
monitor with WXGA (1280 × 800) resolution. They execute
Eclipse 3.7.2 on Windows 7 (Microsoft Corp.) and JDK 1.7.0

385

0.8
0.6
0.4

We operated the experiment three times: once for each organization. For the convenience of the subjects, each operationis conducted at the location where they belong, using the same
instruments and procedure.
At the beginning of the operation, subjects were given
the following introduction in an hour, using a PowerPoint
presentation: (1) the purpose of the experiment and the goal offeature location, (2) usage of Eclipse, e.g. showing hierarchiesof method calls and class inheritance relations, searchingreferences from/to a method, etc., (3) introduction to the
FLPlayer plug-in, (4) an exercise in muCommander using
the example task M0 , and (5) another exercise in jEdit using
the example task J0 .
Ten minutes were given for each example task. Then theanswer and the reason were explained. Subjects were requested
to validate a list of methods so that other developers canunderstand the implementation of the feature. A method isrelevant to a feature if at least a single line of code in themethod is necessary to execute the feature, according to thedefinition of goldsets in the benchmark. Although we initiallyarranged the same configurations of the laptops for all subjects,
they were allowed to change them during the exercises to fit
their preferences. They were told that we close all editors
in Eclipse between sessions, but we do not change theirpreferences settings.
Five sessions were conducted after the introduction. In eachsession, the subjects performed a task in a limited time, 30
min, for avoiding weariness. At the end of the session, they
filled in a questionnaire to answer whether they were able
to understand the task, were given suﬃcient time, and wereconfident with their answers. Between sessions, they wereasked to leave the room and have a break of about 10 min.

BetterWorse

0.0

E. Procedure

0.2

Validated F−measure

1.0

without a network connection. Although each laptop has anembedded pointing device, subjects were allowed to bring theirfavorite devices.

J1

J2

M1

M2

Tasks

Fig. 4. Comparison of validated F-measure between better and worse tasks.
TABLE III
Differences of Accuracy between Initial Lists and Validated Ones
Task
BetterWorse

Δprecision
Mean
S.D.
+0.140 0.230+0.433 0.298

Δrecall
Mean
S.D.
−0.112 0.195+0.145 0.236

ΔF
Mean
S.D.
+0.005 0.231+0.249 0.271

This section presents a discussion of the results of ourexperiment to answer the research questions formulated in
Section III. In this section, initial F-measure and validated
F-measure respectively denote F-measures of the initial listsof tasks given to the subjects and of the resultant lists validated
by the subjects.

we observed that manual validation of better tasks tends tooutperform worse ones.
To determine the performance of two kinds of tasks thatare significantly diﬀerent, we performed paired t-tests on
the validated F-measure. The null hypothesis H0b≤w and thealternative hypothesis H1b>w are formulated as described below.
b≤w• H0 : The average of validated F-measure of better tasks
is equal to or less than that of worse tasks.
b>w• H1 : The average of the validated F-measure of bettertasks is greater than that of worse tasks.
The p-value was 0.008876, which is suﬃciently small toreject H0b≤w at the 1% significance level. We conclude thata statistically significant diﬀerence exist in final F-measure
between better and worse tasks. Subjects given a more accurate
list performed more accurate feature location.
Table III summarized how validation aﬀects precision, recall, and F-measure. All metrics show diﬀerences between
initial lists and validated ones, e.g. ΔF = Fvalidated − Finitial .
The table shows that subjects given a less-accurate list cansignificantly improve the list. In contrast, subjects given an
accurate list tend to decrease recall but keep the overall
accuracy of the list, diﬀerently from the result of a traceability
research [10]. The eﬀect of validation is further discussed in
Section V-D.

A. RQ1B. RQ2In the experiment, each subject performed four tasks, twoof which start with better initial accuracy than the others.
We obtained 40 samples for each of better and worse tasks.
Figure 4 is a box plot of validated F-measure of better andworse tasks. The task pairs are located on the horizontal axis,
where the white and gray boxes respectively correspond tobetter and worse initial F-measure values. The × symbols
indicate the initial F-measure for each task. For all features,

We split samples of better and worse tasks into two categories: precision-intensive and recall-intensive. As a result,
20 samples were obtained for each combination of accuracy
and size of goldset. Figure 5 shows a box plot of validated
F-measure that compares the two categories. In both better
and worse tasks, we observed that tasks with recall-intensive
goldsets tend to outperform those with precision-intensive onesin the validated F-measure. It is noteworthy that our results

V. Results

386

1.0

, : rejection of false and true positives

0.8• , ×: acceptance of true and false positives⊕, +×: addition of false and true negatives
Although final F-measure values varied among subjects,
all obtained final F-measure values were better than those of
initial ones (Finitial = 0.38) within 20 min. All of them found at
least two gold methods in the first 15 min. In addition, sevensubjects (Subjects 3, 4, 5, 10, 12, 13, and 17) selected one
or more false negatives. Two subjects (Subjects 10 and 19)
decreased the F-measures at the end. According to the log,
the subjects seemed to postpone the decision for the methods
that are in doubt about their relevance to the last minute. Thesemethods are used by the feature concerned, but also by otherfeatures. It is noteworthy that the  and ⊕ symbols tend to
appear contiguously, which indicates that when the subjects
find one relevant method, they can correctly distinguish true
positives around the method, and can sometimes even pick up
false negatives.
Similarly, the  symbols tend to appear contiguously in thetime plots. The subjects identified all seven false positives,
except Subject 5, who missed only one. It is noteworthy thatfew  symbols appear in the series, indicating that the subjects
can separate false positives from the list without losing true
positives.
A similar tendency is observed in the time plots of othertasks, but we had to omit them from this paper because oflength limits.

0.4

0.6

•

0.2
0.0

Validated F−measure•

Precision−intensive
Recall−intensive

BetterWorse
Accuracy of Initial Lists

Fig. 5. Comparison of validated F-measure between precision-intensive andrecall-intensive tasks.

show that 12 participants completely identified a feature in one
or two tasks (17 tasks in total). All of them are recall-intensive
tasks.
To determine whether the performance of tasks are significantly diﬀerent or not, we formulated the null hypothesis H0p=r
and the alternative hypothesis H1p<r as shown below.
p=r• H0 : For the tasks with the same level of initial Fmeasure, the median of validated F-measure of precisionintensive tasks is the same as that of recall-intensive ones.
p<r• H1 : For the tasks with the same level of initial Fmeasure, the median of validated F-measure of precisionintensive tasks is less than that of recall-intensive ones.
We performed the Wilcoxon signed-rank test to the medianof validated F-measure for the two categories of tasks. In bothtasks, the p-value was 9.537×10−07 , which is suﬃciently small
to reject H0p≥r at the 1% significance level. We conclude thata statistically significant diﬀerence exists in final F-measure
between the categories. In other words, the initial recall ismore important than the initial precision.
C. RQ3All subjects were able to classify all methods in initial lists
as either relevant or irrelevant in 30 min. We have analyzed
how subjects spent time for their tasks using the timestampsof events recorded in FLPlayer. As a result, no significant
diﬀerence was found among subjects because of their diﬀerent
backgrounds. Also, no significant diﬀerence was found among
tasks. Most subjects finished their tasks in 20 min.
Figure 6 includes time plots showing evolution of Fmeasure values for each subject performing task J2w for
example. The numbers shown indicate the rank of methods inthe initial list. The added methods by subjects have numbers
more than ten. The goldset of this task consists of six methods.
The initial list of methods consists of three true positives andseven false positives. Subjects must find three false negatives
in the source code. Because the median value of the goldset
size in the dataset of Gethers [11] is six, we chose J2w as arepresentative task for locating feature enhancement requests.
The following six kinds of symbols are put on vertices
corresponding to operations of the subjects.

D. RQ4
Figure 7 shows the degrees of improvement in precisionand recall for all treatments. Dashed, solid, and dot-dashlines respectively correspond to better, worse, and goldset
tasks. It is readily apparent that most arrows are moving
rightward, for improving precision, remarkably in worse tasks.
This observation diﬀers from the experiment conducted by
Cuddeback et al. [8], who confirmed a tendency that arrows
move to a diagonal line, precision = recall.
Precision was retained or improved in all tasks by mostsubjects. The subjects created the precise (precision 1.00)
result in 68 of 100 tasks. In 82 tasks, precision is greaterthan 0.8. The result indicates that subjects can exclude many
false positives from the initial lists.
However, recall has not been improved so much. For example, false negatives of M1b and M1w are not identified at
all except for a few subjects. This is true partly because thefeature is implemented by an abstract method and its many
concrete implementations. The abstract method is likely to be
suﬃcient to achieve the feature. The subjects did not identify
all concrete implementations.
In several tasks, many subjects exhibited decreased recall.
In M2b , several subjects falsely recognized one or two relevant
methods as irrelevant. Because each subject has chosen different methods as irrelevant, voting by several subjects caneliminate these errors. However, we have observed that adiﬀerent understanding of a feature easily prevents developersfrom recognizing some true positives as follows.

387

1.0

Subject 3

Subject 5

Subject 7

Subject 10

1 2 6

0.8

10

10

10

3 4 75

9

0.6

12
12
10

9
11

8

5

11

0.4

11

3

9

12345

4
9

11

8

8

8

76

10

9

5

13
910
12 34 56 7

8

0.2

F−measure

Subject 4

13
12

12 6

0.0

123 4 5 6 7

Subject 13

1.0

Subject 120.8

9Subject 17

6

11

0.6
0.4

11

12

8

7

10

34 5
9

9

9

12 3 5 6 7 4

11

9

8

12

8

0.2

8

10

10

345 6710

98

Subject 19

12 345 6 7

1110

F−measure

Subject 14

13
12

0.0

2

12 3

1

0

10

20

30

0

10

Elapsed (min)

20

30

0

10

Elapsed (min)Reject false positive

20

30

0

Elapsed (min)Reject true positive

Accept true positive

4 56

7

10

20

30

Elapsed (min)Accept false positive

Add false negative

0

10

20

30

Elapsed (min)Add true negative

1.0

Fig. 6. Growth of accuracy during manual validation of J2w .

M 2b
J 2b

BetterIn J3 , the task described that the new feature enabledusers to choose file icons for a view from operating system
icons or default icons provided by jEdit. As a result, 13 of
20 subjects recognized several getDefaultIcon methods as
irrelevant because they were likely to have been basic behavior.
However, the methods are involved in the goldset because theconcept of “default icon” has been introduced for the feature.
In other words, the method names reflected the new feature,
whereas their source code implements the basic behavior.
In M1b and M1w , the original description included an
ambiguous phrase “the preferences dialog.” It refers toclass PreferenceDialog and its subclasses GeneralPreferencesDialog and ThemeEditorDialog. Also, 6 of
20 subjects falsely recognized a method of class ThemeEditorDialog as a false positive because the dialog classname diﬀers from the other two classes.

J3

Worse

0.8

GoldsetRecall

0.6

J 1b

0.4

M 2w
J 2w

M 1b

J 1w

0.0

0.2

M 1w

VI. Discussion
0.0

0.2

0.4

0.6

0.8

1.0

Precision

Fig. 7. Change in precision and recall.

In J1b and J3 , many subjects falsely recognized a particulargroup of relevant methods as false positives. The task J1b
asked developers to locate a feature that enabled users tochoose a shape of a marker shown in a window. Whereas theprevious implementation has a triangle marker, the new feature
supported new shapes such as a box and circle. The feature
is implemented as a new item to select a marker shape in apreferences dialog and a new class ShapedFoldPainter for
drawing new marker shapes. However, seven of ten subjects
recognized the methods in the new class as irrelevant, perhaps
because they understood that the feature simply modified thepreferences dialog.

We have shown that a more accurate list of methods enabled
subjects to perform more accurate feature location (RQ1).
That result emphasizes the importance of further research
eﬀorts to improve the accuracy of automated feature location
techniques. Although a tradeoﬀ exists between precision andrecall, developers manually improved precision rather thanrecall (RQ2). As a consequence, a feature location techniquewith higher recall is a more promising direction. In traceability
recovery experiments [8], [9], [10], no significant diﬀerence
was found between precision and recall. One possible reason
for this result is that subjects can understand features in detailduring their tasks by reading relevant methods in initial lists.
Because relevant methods are often connected by method callrelations, more relevant methods in an initial list might provide
a connected call graph that is easier to understand. However, system tests can be independently by natural language.
Therefore, validating a link in a traceability matrix might not388

provide additional information about the manner in which
requirements and system tests are related mutually.
Higher recall might be achieved by a longer list of methods,
as evaluated in [11], [22]. In this experiment, developers
validated 50 methods during five sessions. In each session,developers took about 20 min to validate ten methods. If
automated feature location techniques generate a longer list of
methods, then both learning eﬀects and fatigue might aﬀect
the performance of developers. To utilize a longer list, an
additional support such as a keyword search for a list of
methods would be needed, as Parnin et al. suggested forautomated debugging [18].
Although subjects located a complete implementation of afeature in 17 of 100 tasks, many subjects falsely recognized
relevant methods as false positives. One reason is that atarget feature in a program is dependent on another featurein the program. The goldsets included such methods in thedependent feature because they are also modified to implement the target feature. However, subjects showed diﬃculty
determining whether such methods should be a part of afeature or not. Allowing developers to categorize methods into
three categories as relevant, irrelevant, and marginal might beeﬀective to avoid the problem.
An insuﬃcient description of a feature also prevents developers from accurate feature location. Each description ofa feature request recorded in an issue tracking system often
excludes the basic behavior of software without the feature
because the description is written by users or developers who
know other basic features of the software. To conduct ourexperiment, we must manually extend the description of afeature to enable subjects to distinguish the target feature frombasic features.
A clearer understandable description of a feature is important but challenging because a description might become
ambiguous if developers modified source code. For example,
one presumes a description that explicitly refers to classes
by their names, e.g. PreferencesDialog instead of a phrase“the preferences dialog.” If a developer added a new subclassof the class, the developer must inspect the description andthe feature implementation to ascertain whether the description
should be updated or not. This problem is similar to fragile
pointcut problem [23]. A pointcut is a predicate to identify
program elements in Aspect-Oriented Programming [24]. A
pointcut is fragile because a change in source code accidentallyaﬀects a set of program elements selected by the pointcut. A
feature description might be more fragile because it is written
in natural language, whereas pointcut is a formal predicate.
As Mäder et al. [25] proposed for analysis of developers’ activities aﬀecting traceability links, some technique to maintainconsistency between feature description and source code might
be very useful.
Another means to improve recall is the use of interactive
feature location techniques such as [26], [27]. They canrecommend methods related to a method by which developers
focus so that developers can identify more false negatives insource code. If the tools can provide an explanation of why

methods are recommended as relevant, then the tools might
be eﬀective to avoid the accidental exclusion of true positives.
VII. Threats to Validity
A. Internal Validity
The first threat to internal validity is related to the feature
location technique we used for the experiment. To obtainlists of methods and to control their accuracy artificially,
we exploit the result of LSI, an IR-based feature location
technique taken from the dataset of Gethers et al. [11]. Whilefalse positive methods selected by LSI includes keywords
semantically related to features, they might be easier to be
identified because developers could investigate the methods
using diﬀerent criteria such as method call relationship anddata dependence. A diﬀerent result might be observed if
goldsets and false positives are selected by those diﬀerent
criteria.
Because comparison of feature location techniques by precision and recall is a common means of evaluation in the literature, we expect that the IR-based technique can be replacedwith another as long as its result has comparable precisionand recall. However, given a feature location technique forprioritizing methods that are diﬃcult for developers to find,then the result might be inverted. For example, De Luciaet al. [28] conducted an experiment on labeling classes andpointed out that IR-based techniques can find informationthat is diﬃcult for developers to find. Similar eﬀects mightoccur in IR-based feature location techniques. In fact, somefeature location techniques recommend relevant methods froman initial list [26], [27]. These techniques are consistent withthe behavior of the subjects observed in RQ3. They prevent
developers from missing false negatives. If these techniques
are used in the experiment, then manual validation might work
better in adding false positives rather than removing false
negatives.
Another internal validity is a concern about the feature location tasks we designed. To avoid feature location tasks being
overly dependent on a system, we choose feature requests fromtwo systems. Additionally, we took two features from eachsystem: one feature has a larger goldset; the other has smaller
one, to avoid dependence to the size of the goldset. The task
we prepared showed better and worse results for each of thefour feature requests, and one task which gives the goldset
itself. The result of our experiment might be dependent on
those tasks. This threat can be decreased by adding more tasks
and by replicating the experiment because we also make thedataset available online.
B. External Validity
As for the external validity, we believe that the results ofour experiment can be generalized for use in other academic
and industrial organizations. We recruited subjects from two
universities and an industrial company and asked them toperform the same tasks. Because all of their organizations
are specialized in software engineering, a weak threat exists
by which the subjects might share some background in the389

discipline. We regard this threat as acceptable because theirspectrum of programming experience ranges from a few to 40
years.
We also believe that the result can be generalized to othersystems written in Java language. However, the result might
not be applicable in industry because the tasks are taken only
from open-sourced systems.
C. Construct Validity
The major threat related to construct validity is that weused F-measure to assess the accuracy of both initial lists
and validated lists. Although the F-measure captured the total
improvement of precision and recall, diﬀerent precision andrecall values might result in the same F-measure. To avoid
that problem, we analyzed precision and recall separately in
Section V-D. However, a threat still remains. If subjects often
excluded a true positive from a list and included a false
negative in the list, then the recall value is not changed. Suchan eﬀect is not readily apparent in the metric.
VIII. Conclusion
As described in this paper, we have conducted a controlled
experiment of feature location tasks. We have prepared listsof methods obtained using an automated technique, but theiraccuracy is controlled artificially. We asked 20 subjects tovalidate the lists manually. Consequently, the validated lists
were totally improved from the initial lists. Developers could
improve precision by recognizing false positives well, but they
could not improve recall. This is true because a diﬀerent
understanding of a feature prevented the developers fromrecognizing true positives.
Several avenues of future eﬀort remain. Due to reducing
time eﬀort for subjects, we kept the lists of methods small.
How large a list of methods developers can reject false positives from the list should be investigated. Another importantquestion is whether an incomplete feature location result isalso eﬀective for maintenance tasks, as similar to the eﬀectof full requirements-to-code traceability reported by Mäder et
al. [2]. Because developers understand more about a featureduring their maintenance tasks, a partially located featuremight be suﬃcient for developers. We are also interested in
improving the feature location benchmark. We expect thatthe curation of goldsets described in Section IV-A can be
automated to some extent by tracing changes on gold methods
in the version history.
Acknowledgments. We would like to thank all the subjects
who participated in this study. This work was supported by
KAKENHI (Nos. 23680001, 23700030, and 25220003).
References[1] B. Dit, M. Revelle, M. Gethers, and D. Poshyvanyk, “Feature location
in source code: A taxonomy and survey,” J. Softw.: Evol. and Proc.,vol. 25, no. 1, pp. 53–95, 2013.
[2] P. Mäder and A. Egyed, “Assessing the eﬀect of requirements traceability
for software maintenance,” in Proc. ICSM, 2012, pp. 171–180.
[3] J. Wang, X. Peng, Z. Xing, and W. Zhao, “An exploratory studyof feature location process: Distinct phases, recurring patterns, andelementary actions,” in Proc. ICSM, 2011, pp. 213–222.

[4] A. Egyed, F. Graf, and P. Grünbacher, “Eﬀort and quality of recovering
requirements-to-code traces: Two exploratory experiments,” in Proc. RE,
2010, pp. 221–230.
[5] A. Marcus, A. Sergeyev, V. Rajlich, and J. I. Maletic, “An information
retrieval approach to concept location in source code,” in Proc. WCRE,
2004, pp. 214–223.
[6] D. Poshyvanyk, Y.-G. Guéhéneuc, A. Marcus, G. Antoniol, and V. Rajlich, “Feature location using probabilistic ranking of methods based on
execution scenarios and information retrieval,” IEEE Trans. Softw. Eng.,
vol. 33, no. 6, pp. 420–432, 2007.
[7] M. Eaddy, A. V. Aho, G. Antoniol, and Y.-G. Guéhéneuc, “CERBERUS:
Tracing requirements to source code using information retrieval, dynamic analysis, and program analysis,” in Proc. ICPC, 2008, pp. 53–62.
[8] D. Cuddeback, A. Dekhtyar, and J. H. Hayes, “Automated requirements
traceability: The study of human analysts,” in Proc. RE, 2010, pp. 231–
240.
[9] W.-K. Kong, J. H. Hayes, A. Dekhtyar, and J. Holden, “How do we trace
requirements? an initial study of analyst behavior in trace validation
tasks,” in Proc. CHASE, 2011, pp. 32–39.
[10] A. Dekhtyar, O. Dekhtyar, J. Holden, J. H. Hayes, D. Cuddeback, and
W.-K. Kong, “On human analyst performance in assisted requirements
tracing: Statistical analysis,” in Proc. RE, 2011, pp. 111–120.
[11] M. Gethers, B. Dit, H. Kagdi, and D. Poshyvanyk, “Integrated impact
analysis for managing software changes,” in Proc. ICSE, 2012, pp. 430–
440.
[12] A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, “An exploratory
study of how developers seek, relate, and collect relevant information
during software maintenance tasks,” IEEE Trans. Softw. Eng., vol. 32,
no. 12, pp. 971–987, 2006.
[13] M. Lindvall and K. Sandahl, “How well do experienced softwaredevelopers predict software change?” J. Syst. Softw., vol. 43, no. 1, pp.
19–27, 1998.
[14] W. Zhao, L. Zhang, Y. Liu, J. Sun, and F. Yang, “SNIAFL: Towards astatic noninteractive approach to feature location,” ACM Trans. Softw.
Eng. Methodol., vol. 15, no. 2, pp. 195–226, 2006.
[15] M. Revelle and D. Poshyvanyk, “An exploratory study on assessing
feature location techniques,” in Proc. ICPC, 2009, pp. 218–222.
[16] D. Cuddeback, A. Dekhtyar, J. H. Hayes, J. Holden, and W.-K. Kong,“Towards overcoming human analyst fallibility in the requirements
tracing process,” in Proc. ICSE, 2011, pp. 860–863.
[17] A. Ghabi and A. Egyed, “Code patterns for automatically validating
requirements-to-code traces,” in Proc. ASE, 2012, pp. 200–209.
[18] C. Parnin and A. Orso, “Are automated debugging techniques actually
helping programmers?” in Proc. ISSTA, 2011, pp. 199–209.
[19] D. Chatterji, J. C. Carver, B. Massengill, J. Oslin, and N. A. Kraft,“Measuring the eﬃcacy of code clone information in a bug localization
task: An empirical study,” in Proc. ESEM, 2011, pp. 20–29.
[20] D. Binkley, D. Lawrie, and C. Uehlinger, “Vocabulary normalization
improves IR-based concept location,” in Proc. ICSM, 2012, pp. 588–
591.
[21] M. Beard, N. Kraft, L. Etzkorn, and S. Lukins, “Measuring the accuracyof information retrieval based bug localization techniques,” in Proc.
WCRE, 2011, pp. 124–128.
[22] A. D. Eisenberg and K. D. Volder, “Dynamic feature traces: Finding
features in unfamiliar code,” in Proc. ICSM, 2005, pp. 337–346.
[23] C. Koppen and M. Störzer, “PCDiﬀ: Attacking the fragile pointcut
problem,” in Proc. European Interactive Workshop on Aspects in
Software, 2004. [Online]. Available: http://pp.info.uni-karlsruhe.de/
uploads/publikationen/stoerzer04eiwas.pdf[24] G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. V. Lopes,
J. Loingtier, and J. Irwin, “Aspect oriented programming,” in Proc.
ECOOP, 1997, pp. 220–242.
[25] P. Mäder, O. Gotel, and I. Philippow, “Enabling automated traceability
maintenance by recognizing development activities applied to models,”in Proc. ASE, 2008, pp. 49–58.
[26] M. P. Robillard, “Automatic generation of suggestions for program
investigation,” in Proc. ESEC/FSE, 2005, pp. 11–20.
[27] M. Trifu, “Improving the dataflow-based concern identification approach,” in Proc. CSMR, 2009, pp. 109–118.
[28] A. De Lucia, M. Di Penta, R. Oliveto, A. Panichella, and S. Panichella,“Using IR methods for labeling source code artifacts: Is it worthwhile?”in Proc. ICPC, 2012, pp. 193–202.

390