An Unsupervised Approach for Discovering Relevant Tutorial Fragments for APIs
He Jiang1, 2, 3
jianghe@dlut.edu.cn
2

Jingxuan Zhang1

Zhilei Ren1

Tao Zhang4

jingxuanzhang@mail.dlut.edu.cn

zren@dlut.edu.cn

cstzhang@hrbeu.edu.cn

1School of Software, Dalian University of Technology, Dalian, China
Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, Dalian, China
3
State Key Laboratory of Software Engineering, Wuhan University, Wuhan, China
4
College of Computer Science and Technology, Harbin Engineering University, Harbin, China

Abstract—Developers increasingly rely on API tutorials tofacilitate software development. However, it remains achallenging task for them to discover relevant API tutorial
fragments explaining unfamiliar APIs. Existing supervised
approaches suffer from the heavy burden of manually
preparing corpus-specific annotated data and features. In thisstudy, we propose a novel unsupervised approach, namely
Fragment Recommender for APIs with PageRank and Topic
model (FRAPT). FRAPT can well address two main challenges
lying in the task and effectively determine relevant tutorial
fragments for APIs. In FRAPT, a Fragment Parser is proposed
to identify APIs in tutorial fragments and replace ambiguous
pronouns and variables with related ontologies and API
names, so as to address the pronoun and variable resolution
challenge. Then, a Fragment Filter employs a set of nonexplanatory detection rules to remove non-explanatory
fragments, thus address the non-explanatory fragment
identification challenge. Finally, two correlation scores areachieved and aggregated to determine relevant fragments forAPIs, by applying both topic model and PageRank algorithm
to the retained fragments. Extensive experiments over two
publicly open tutorial corpora show that, FRAPT improves thestate-of-the-art approach by 8.77% and 12.32% respectively interms of F-Measure. The effectiveness of key components of
FRAPT is also validated.
Keywords-Application Programming Interface; PageRank
Algorithm; Topic Model; Unsupervised Approaches

I.

INTRODUCTION

Reusing Application Programming Interfaces (APIs) inexisting libraries and frameworks could greatly speed up
software development process [1, 2]. Hence, it is critical fordevelopers to search and use APIs properly [3]. As one of themost important API documentations, API tutorials are often
consulted by developers to learn how to use APIs in a given
programming context [4, 5]. However, API tutorials areusually lengthy and mixed with irrelevant information, so it
is tedious for developers to peruse a full tutorial for an
unfamiliar API [6]. One feasible solution is to split tutorials
into consecutive fragments and recommend relevant
fragments containing explanation information to developers[3, 7]. Hereafter, if a fragment explains an API, then they are
relevant. Otherwise, they are irrelevant.
In recent years, three approaches have been proposed toresolve the task of discovering relevant tutorial fragments forAPIs in the literature, namely FITSEA [7], GMR [3], and IR[3]. Among these approaches, FITSEA and GMR aresupervised approaches and dramatically outperform IR.
There are two stages in these supervised approaches, namely
training and test. In the training stage, tutorials are split intofragments and APIs are manually labeled as relevant or
irrelevant to their fragments. The labeled APIs and theirfragments form a series of fragment-API pairs. Somepredefined features are extracted from each fragment-API
pair. These extracted features with their class labels (relevant
or irrelevant) are taken to train a classifier. In the test stage,
the trained classifier is used to predict the class labels of new
fragment-API pairs.
However, there are some drawbacks in these supervised
approaches, due to the heavy burden of manual annotation
efforts and feature construction.
 Different corpora require their corpus-specificannotated data, thus cost a lot of manual efforts. Supervised
approaches may easily lead to the bias between training andtest, especially when conducting cross-corpus prediction [3].
 The effectiveness of supervised approaches largely
depends on the features to capture the specific characteristics
of corpora. When applying the constructed features on
different corpora, classifiers may behave well on some
corpora, while poorly on the others [3, 7].
The above drawbacks in supervised approaches motivate
us to propose an unsupervised approach, which possesses
several advantages compared to supervised approaches [8, 9].
First, there is no need to annotate corpus-specific data andtrain a classifier, so that it costs far less manual efforts andavoids the bias between training and test. Second,
unsupervised approaches can adapt to different corpora byadjusting some parameters instead of constructing features.
In this study, we propose a novel unsupervised approach,namely Fragment Recommender for APIs with PageRank
and Topic model (FRAPT). FRAPT can find the relevance
between fragments and APIs effectively and recommend
relevant fragments for APIs to developers automatically. In
FRAPT, all fragments are taken into a Fragment Parser to
identify APIs, replace ambiguous pronouns and variables
with related ontologies and API names, and find sentenceboundaries and their types after tutorial segmentation. Then,a Fragment Filter is introduced to filter out non-explanatory
fragments which do not explain any APIs. Next, both topic
model and PageRank algorithm are applied to each retainedfragment to achieve two types of correlation scores between
fragments and APIs. By aggregating the correlation scores,
relevant fragments for APIs can be identified andrecommended for developers.
To evaluate the effectiveness of FRAPT and validate theimpacts of its key components, we conduct extensive
experiments on all publicly available annotated tutorial
corpora, namely McGill corpus [38] and Android corpus [39].
FRAPT improves the state-of-the-art approach FITSEA by
8.77% and 12.32% respectively in terms of F-Measure. In
addition, the key components of FRAPT also exhibit theireffectiveness. For example, by aggregating the correlation
scores from both topic model and PageRank algorithm,
better F-Measure can be achieved and improve any of themby up to 9.10% and 4.88% over the two tutorial corpora.
The main contributions of this study are listed as follows:(1) A Fragment Filter with a set of non-explanatory
detection rules is proposed to remove non-explanatory
fragments effectively.
(2) We propose a new unsupervised approach FRAPT
leveraging topic model and PageRank algorithm to discover
relevant fragments for APIs. FRAPT is publicly available at
http://oscar-lab.org/people/~jxzhang/FRAPT/.
(3) A series of experiments are conducted to validate theeffectiveness of FRAPT and its components. The results
demonstrate that FRAPT significantly improves the state-ofthe-art supervised approach.
II.

MOTIVATION EXAMPLESIn this section, we discuss the main challenges lying in
the task of discovering relevant fragments for APIs.
APIs can be explained at different levels by tutorials,
ranging from package level, class or interface level, tomethod level. In order to facilitate comparison of different
approaches, similar as [3, 7], APIs are chosen at class orinterface level in this study. On one hand, when facing an
unfamiliar API, developers usually want to know how to usesome available behaviors rather only one method [3]. On theother hand, tutorials usually introduce some programming
topics by using a set of methods of classes [7]. As a result,
APIs are selected at class or interface level in this study.
Fig. 1 shows two fragment examples. Fragment A andFragment B are extracted from the JodaTime tutorial [40]and the Android Graphics tutorial [41] respectively.
Fragment A has three paragraphs with four APIs, namely
DateTime, Interval, Duration, and Period. However,Fragment A is irrelevant to any of them according to themanual annotation [40]. In contrast, there are two paragraphswith a piece of code example in Fragment B. Four APIs aredetected, namely Canvas, Bitmap, SurfaceHolder, andSurfaceView. According to the manual annotation [41],Canvas and Bitmap are relevant to this fragment, whereas
SurfaceHolder and SurfaceView are not.
Based on the above tutorial fragments, we find that two
challenges need to be addressed for effectively discovering
relevant tutorial fragments for APIs.
 Pronoun and variable resolution
Pronouns and variables are widely used in API tutorials.
These pronouns and variables may be ambiguous, if we do(1)JodaTime is like an iceberg, 9/10ths of it is invisible to user-code. (2)Many,
perhaps most, applications will never need to see what's below the surface.
(3)This document provides an introduction to the JodaTime API for the average
user, not for the would-be API developer.
(4)The bulk of the text is devoted to code snippets that display the most common
usage scenarios in which the library classes are used. (5)In particular, we cover
the usage of the key DateTime, Interval, Duration and Period classes.
(6)We finish with a look at the important topic of formatting and parsing and a
few more advanced topics.

(a) Fragment A(1)When you're writing an application in which you would like to perform
specialized drawing and/or control the animation of graphics, you should do so by
drawing through a Canvas. (2)A Canvas works for you as a pretense, or interface,to the actual surface upon which your graphics will be drawn. (3)It holds all of
your “draw” calls. (4)Via the Canvas, your drawing is actually performed upon anunderlying Bitmap, which is placed into the window.
(5)In the event that you're drawing within the onDraw() callback method, theCanvas is provided for you and you need only place your drawing calls upon it.
(6)You can also acquire a Canvas from SurfaceHolder.lockCanvas(), when dealing
with a SurfaceView object (Both of these scenarios are discussed in the following
sections). (7)However, if you need to create a new Canvas, then you must define theBitmap upon which drawing will actually be performed. (8)The Bitmap is always
required for a Canvas. (9)You can set up a new Canvas like this:(10) Bitmap b = Bitmap.createBitmap(100, 100, Bitmap.Config.ARGB_8888);(11) Canvas c = new Canvas(b);(b) Fragment B
Figure 1. Fragment examples

not consider their surrounding sentences or programming
context. We observe that about 70% fragments contain atleast one pronoun and 50% fragments with code examples
declare variables in the open tutorial corpora [38, 39]. The
influences of APIs will be weakened, if the ontologies ofthese pronouns and variables are APIs.
Taking Fragment B from the Graphics tutorial [41] in Fig.
1 as an example, if we only consider the third sentence “Itholds all of your draw calls”, the ontology of “It” isambiguous. With the help of its context, we can infer that “It”
stands for the API Canvas. In contrast, the second statement
in code example in Fragment B declares a Canvas variable cwith b as a parameter. By inspecting its programming
context, we can infer that b stands for the API Bitmap.
 Non-explanatory fragment identification
Not all the fragments are designed to explain APIs. Nonexplanatory fragments do not explain any APIs (accordingly,
an explanatory fragment explains at least one API). As to an
investigation on the tutorial corpora used in this study [38,39], we find that 10% ~ 50% fragments are annotated as nonexplanatory fragments.
For example, Fragment A belongs to non-explanatory
fragments, since all of its APIs appear in an enumeration
sentence, and Fragment A just gives an overview of thewhole JodaTime tutorial [40].
In the following part of this paper, we present how our
new approach FRAPT addresses the above challenges andexplain its workflow with Fragment A and Fragment B.
III.

FRAMEWORK OF FRAPTThe whole framework of FRAPT consists of two phases,
namely the relevance discovery phase and the fragment
recommendation phase (see Fig. 2).
Relevance Discovery Phase. This phase aims to findthe relevance between fragments and APIs. FRAPT firstUnfamiliar
APIs

Tutorials

Relevant
Fragments for
Unfamiliar APIs
2

1
Tutorial
Segmentation
Fragments
Fragment
ParserAPI
Discovery

Pronoun &Variable
Resolution

SentenceType
Identification

Sentence
Identification

Topic Based
Correlation Score
Calculation

ParsedFragments

Relevant
Fragment
Recommender

Topic Based
ScoresFragment
Filter1

2

Retained
Fragments
PageRank Based
Correlation Score
Relevance Discovery Phase
Calculation

Relevance
Betweenfragments
and APIs

Relevance
Identification
PageRank
Based Scores

Fragment Recommendation Phase

FRAPT

Figure 2. Workflow of FRAPT

segments tutorials into coherent fragments by a Tutorial
Segmentation component, and inputs the fragments into a
Fragment Parser to detect APIs, resolve pronouns insentences and variables in code examples to address the
pronoun and variable resolution challenge, and identify
sentences with their types. Then, the parsed fragments aretaken into a Fragment Filter to detect and filter out nonexplanatory fragments, so as to address the non-explanatory
fragment identification challenge. Next, on one hand, thefragments are taken into a topic model to find semantic
connections between fragments and APIs. Based on the
fragment-topic matrix and the topic-term matrix obtained
from the topic model, a type of correlation scores between
fragments and APIs can be achieved. On the other hand,
PageRank algorithm is applied to the graph constructed bysentences in fragments, and another type of correlation
scores between fragments and APIs can also be achieved
according to the PageRank values of sentences containing
APIs. Last, a relevance identification scheme is designed todecide the relevance between fragments and APIs.
Fragment Recommendation Phase. The goal of thisphase is to recommend relevant fragments for the unfamiliar
APIs queried by developers. When a developer wants to use
an unfamiliar API to accomplish some programming tasks,
he/she inputs the unfamiliar API into FRAPT. FRAPT looks
up the relevance between fragments and APIs obtained fromthe first phase, and recommends relevant tutorial fragments
to the developers.
IV.

MAIN COMPONENTS OF FRAPT

A. Tutorial Segmentation
Tutorial segmentation follows the same method as [3, 7]
to divide tutorials into coherent fragments. First, we split
tutorials into a series of paragraphs. Then, we iteratively
merge sibling paragraphs until the length of the paragraphs
reaches within a specified range (100 words to 300 words).
In such a way, the tutorial fragments can be achieved foreach tutorial.

B. Fragment Parser
Fragment Parser conducts four steps to parse a fragment,
namely API Discovery, Pronoun and Variable Resolution,
Sentence Identification, and Sentence Type Identification.
(1) API Discovery
First of all, we detect all the APIs contained in eachfragment. All the open tutorials are available on the Internetin the form of HTML. Although APIs in different tutorials
have different styles, most of tutorials follow W3C
guidelines and link APIs to their corresponding specification
webpages with <href> tags. By parsing the linked webpages
and matching API names with the anchor text, we can decide
whether a link is an API or not.
Some tutorials do not link APIs to their specification
webpages. In this situation, we tokenize the tutorials into a
stream of tokens and match these tokens with all theavailable API names. The same as [19], if a token matches
an API name lexically, we continue to search its context, asequence of tokens surround this token with a window sizeof 2, to find whether it contains some keywords, such as“class”, “interface”, and “API”. If yes, it is treated as an API.
With API discovery, four APIs are detected for eachfragment in Fig. 1. DateTime, Interval, Duration, and Period
are discovered in Fragment A. Canvas, Bitmap,
SurfaceHolder, and SurfaceView are detected in Fragment B.
(2) Pronoun and Variable Resolution
To address the pronoun and variable resolution
challenge, Pronoun and Variable Resolution identifies
ontologies for pronouns and APIs for variables, and replacespronouns and variables with their ontologies and APIs.
In FRAPT, Reconcile [10], an automatic coreference
resolution tool, is leveraged to perform pronoun resolution,
thus we can obtain mappings between pronouns and their
ontologies. We replace all the pronouns with their ontologies
for all the sentences. In contrast, if a variable is declared in apiece of code example and some operations are performed on
the variable in the following statements, the variable is used
to call its methods rather than its API, especially in objectoriented programming languages. We inspect each statementin the code examples, and build a mapping between eachvariable and its API. Then we replace all the variables with
their APIs for each statement. In such a way, the impacts ofAPIs are enhanced.
Taking Fragment B as an example, the third sentence “Itholds all of your ‘draw’ calls.” in the first paragraph starts
with a pronoun, namely “It”. By analyzing the context of thissentence, Reconcile [10] finds that “It” stands for the API
Canvas. As a result, we replace “It” with Canvas so that the
API Canvas is enhanced. There are two code statements inthis fragment. The first statement creates a Bitmap variable b,
and the second statement declares a Canvas variable c usingb as a parameter. By analyzing the statements, we can create
two mappings, namely <b, Bitmap> and <c, Canvas>.
Variable b appears in the second statement, and we replace itwith its API, namely Bitmap. As a result, the two statements
are changed to:(10) Bitmap b = Bitmap.createBitmap(100, 100, Bitmap.Config.ARGB_8888);(11) Canvas c = new Canvas(Bitmap);

(3) Sentence Identification
Tutorials usually contain not only API explanations, butalso code examples to show how APIs are used in specificsituations. To obtain each sentence, we delete all the HTMLtags in fragments. An open tool, namely LingPipe [11] isleveraged to detect the sentence boundaries so that thesentences are identified. We treat each statement in code
examples as one sentence. We can obtain statements bysplitting at the places of semicolons. However, there aresome exceptions, e.g., “if” and “for” statements. For the
exceptions, we can split code at this place where there is amatch between left parenthesis and right parenthesis.
We identify 6 and 11 sentences for Fragment A andFragment B respectively, and mark each sentence with asentence ID (see Fig. 1).
(4) Sentence Type Identification
Different types of sentences have different contributions
to explain APIs. To differentiate the importance of eachsentence, we define two sentence types, i.e. marginal
sentences and principal sentences. Marginal sentences
include conditional sentences, enumeration sentences,
example sentences, comparative sentences, and code
comments. Principal sentences are the other sentences except
the above sentences. Intuitively, principal sentences are more
important than marginal sentences. Marginal sentences canbe detected by keyword matching in the following ways:
 Conditional sentences can be found by keywords of“if” and “whether”.
 Enumeration sentences can be detected, when morethan two APIs are listed and connected by “and” or “or”.
 Example sentences can be detected by somekeywords, e.g., “for example”, “for instance”, and “such as”.
 Comparative sentences can be detected by keywords
such as “compared to”, “unlike…”, and “more…than…”. In
this study, inheritance-describing sentences are also treatedas comparative sentences, which can be found by keywords,
e.g., “extend”, “inherited from”, and “subtype of”.
 Code comments start with the string of “//”, “/*” etc.
The fifth sentence in Fragment A is an enumeration
sentence and the seventh sentence in Fragment B is aconditional sentence. As a result, they are marginal sentences.
The other sentences can be viewed as principal sentences.
C. Fragment Filter
Fragment Filter aims to filter out those non-explanatory
fragments which do not explain any APIs by some nonexplanatory detection rules. We first specify how we perform
an in-depth observation to identify these non-explanatory
detection rules. Then we show the details of the rules.
(1) Non-explanatory Detection Rules Identification
There are three steps to identify non-explanatory
detection rules, namely non-explanatory fragment
identification, characteristics formulation, and nonexplanatory detection rules generation.
Non-explanatory Fragment Identification. We selectthe Essential Java Classes tutorials as the representative
tutorials to observe, since they introduce basic Java classes(e.g., exception handing and I/O processing), and do not
overlap with the two tutorial corpora used in this study [37].

Therefore, the identified non-explanatory detection rules canbe applied to the tutorial corpora in this study without
introducing a sampling bias into the results. We segment the
Essential Java Classes tutorials into fragments by the
Tutorial Segmentation component, and evenly distributethese fragments to the second and the third author of thisstudy to check. They are required to analyze and understandthese fragments to select the non-explanatory fragments.
Characteristics Formulation. For each identified nonexplanatory fragment, the two authors investigate what are
the characteristics that lead this fragment to be a nonexplanatory fragment. To find the answer, the authors
measure different characteristics for each non-explanatory
fragment, and check if one or some of them could answer
this question, e.g., the number of sentences and the numberof contained APIs. If they convince that one or somecharacteristics of a fragment make it to be a non-explanatory
fragment, they formulate a new characteristic or merge it toan existing one. Finally, the authors achieve two sets offormulated characteristics.
Non-explanatory Detection Rules Generation. The two
authors check the two formulated characteristic sets to findthe common characteristics appearing in both two sets. Interrater Kappa agreement is applied to evaluate the two
formulated characteristic sets [36]. The Kappa agreement is
0.46 showing a moderate agreement. Then, they decide themost suitable threshold for each common characteristic
found by two authors simultaneously. For example, howmany sentences can differentiate non-explanatory fragmentsfrom explanatory fragments. In such a way, throughsuccessive iterations of discussion and coordination, theyfind some characteristics that could well detect the boundary
between non-explanatory fragments and explanatory
fragments. These characteristics with their thresholds aretransformed to some non-explanatory detection rules.
(2) Non-explanatory Detection Rules
After the non-explanatory detection rules identification,
we find that non-explanatory fragments comply with at least
one of the following characteristics (rules): Non-explanatory fragments usually contain only
one API, and this API appears only once.
The more times an API appears in a fragment, the morechances the fragment is relevant to this API. The fragment
pays less attention on API explanation, if there is only one
API.
 The lengths of non-explanatory fragments areusually less than five sentences.
In such a short fragment, an API cannot be fully
explained, considering that most of APIs may contain morethan one method so as to have sophisticated behaviors.
 The proportion of sentences containing APIs is lessthan 20% in non-explanatory fragments.
It is hard to explain any API in such a small proportion ofsentences containing APIs.
 All the APIs in non-explanatory fragments only
appear in marginal sentences.
In this situation, APIs are only used to be listed as
examples or enumeration. In the other principal sentences,
the fragments focus on other information rather than APIs.

Table I. API PROBABILITIES ASSIGNED TO EACH TOPIC
Term (API) Topic 1
canvas0.0002
bitmap0.0000surfaceholder 0.0001
surfaceview 0.0023

Topic 20.03600.00000.04810.0336

Topic 30.00000.00000.00000.0000Topic 40.42700.33600.00540.0005Topic 50.00000.00050.00000.0000TABLE II. SIMILARITY MATRIX OF DIRECTED GRAPH FOR FRAGMENT B
…
…
…
…
…

 Non-explanatory fragments have some good
indicator terms and phrases.
Some keywords and phrases can give important clues andact as good indicators for finding non-explanatory fragments,
e.g., “summary”, “overview”, “introduction”, “for more
information about …, see ...”, and “more details in …”.
Based on these characteristics, we construct a Fragment
Filter with five non-explanatory detection rules to identify
non-explanatory fragments. If one fragment conforms to anyof these rules, it is marked as a non-explanatory fragment tobe filtered out.
In Fragment A, all the APIs appear in an enumeration
sentence, so it follows the fourth rule. As a result, Fragment
A is detected as a non-explanatory fragment and filtered out.
In contrast, Fragment B is inconsistent with none of theserules and requires further processing.
D. Topic Based Correlation Score Calculation
Topic based correlation score calculation aims to achievea correlation score between fragments and APIs according to
the results of topic model. As a popular way to analyze alarge scale of documents, topic model can be used to findsemantic relationships between documents and terms [12,13]. As a result, each document is expressed by a series of
topics with different probabilities. In contrast, a topic isrepresented by a collection of terms with various
probabilities. In this study, one type of topic model, namely
Latent Dirichlet Allocation (LDA) is leveraged, and Stanford
Topic Modeling Toolbox [14] is introduced to help us toperform LDA. We introduce the same method as [12] toobtain the best configurations (i.e., the topic number) for
LDA. Each tutorial fragment is treated as a document to put
into LDA, and LDA outputs the topic probability
distributions for each fragment as well as the termprobability distributions for each topic. Here, the terms
include not only pure text, but also APIs. Since we intend toinvestigate the correlation scores between fragments andAPIs, only the API probability distributions for all the topics
are used in the following part.
Based on the output of LDA, we can obtain a correlation
score between fragments and APIs by the following formula:
Score (API, fragment) = ∑t∈topic P API | t ×P (t | fragment)(1)where P (API | t) is the API probability for topic t, andP (t | fragment) is the probability of topic t for the fragment.
Taking Fragment B as an example, the fragment can be
represented by five topics with non-zero probabilities,
namely {0.04, 0.01, 0.02, 0.92, 0.01}. Table I shows the API
probability distributions on the five topics. Following theformula (1), the ScoreT values for Canvas, Bitmap,
SurfaceHolder, and SurfaceView with Fragment B are
0.39321, 0.30913, 0.00545, and 0.00089 respectively.

(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
ID (1)
0.1573 0.1759 0.2580 0.2018 0.0706 0.3419 0.1207 0.2060 0.0000 0.2006(1)(2) 0.1573 - 0.1015 0.0766 0.0782 0.1465 0.1015 0.1252 0.2137 0.0000 0.2081(3) 0.1759 0.1015 - 0.1628 0.3227 0.1058 0.2157 0.1808 0.3086 0.0000 0.3005(4) 0.2580 0.0766 0.1628 - 0.1697 0.0799 0.5480 0.3802 0.2330 0.2423 0.4660(5) 0.2018 0.0782 0.3227 0.1697 - 0.0815 0.2249 0.1394 0.2379 0.0000 0.2317(6) 0.0706 0.1465 0.1058 0.0799 0.0815 - 0.1058 0.1306 0.2227 0.0000 0.2169(7) 0.3419 0.1015 0.2157 0.5480 0.2249 0.1058 - 0.5038 0.3087 0.3864 0.6175(8) 0.1207 0.1252 0.1808 0.3802 0.1394 0.1306 0.5038 - 0.3809 0.3961 0.7618(9) 0.2060 0.2137 0.3086 0.2330 0.2379 0.2227 0.3087 0.3809 - 0.0000 0.6329(10) 0.0000 0.0000 0.0000 0.2423 0.0000 0.0000 0.3864 0.3961 0.0000 - 0.3887(11) 0.2006 0.2081 0.3005 0.4660 0.2317 0.2169 0.6175 0.7618 0.6329 0.3887 PR 0.8018 0.6062 0.8563 1.1264 0.7846 0.5857 1.4106 1.3163 1.1869 0.6559 1.6694

E. PageRank Based Correlation Score Calculation
This component aims to obtain another correlation score
between fragments and APIs based on the PageRank value ofeach sentence. PageRank algorithm attempts to evaluate theimportance of each sentence through link analysis [15, 16].
First of all, we create a directed graph based on the sentences
in a fragment, and each vertice in the graph stands for asentence. To build directed edges between vertices, we
calculate the widely used cosine similarity with TF-IDF term
weight between sentences [7]. If the similarity is not zero
between two sentences, we draw two directed edges toward
each other between their corresponding vertices. In such away, we can build a directed graph. After PageRankalgorithm is applied to the directed graph, we can obtain the
PageRank value for each sentence.
Different sentences have different contributions to
explain APIs. The greater the PageRank value of a sentence,
the more closely it catches the central meaning of thefragment. If an API appears in sentences with greater
PageRank values, then it is more likely to be explained bythis fragment. A scheme is designed to obtain correlation
scores between fragments and APIs as follows:
Score

API, fragment =∑s∈fragment PR (s) × P (API | s)∑s∈fragment PR (s)(2)

where s stands for a sentence in the fragment, PR (s) means
the PageRank value of sentence s, andP (API | s) =1
0

if s contains API
if s does not contain API(3)

Table II shows the matrix representation of the directed
graph with PageRank values listed in the last line for
Fragment B. The IDs are their corresponding sentence ID,
and the values in the table show the similarities between thecorresponding sentences. The PageRank values range from0.5857 to 1.6694 for each sentence. The API Canvas
appears in ten sentences. Note that Canvas has alreadyreplaced “It” in the sentence “It holds all of your ‘draw’
calls.” after pronoun resolution. The ScorePR between
Canvas and Fragment B is 0.9404 according to the formula(2). Bitmap appears five times, since the variable b in thelast statement has been replaced by Bitmap. As a result, theScorePR between Bitmap and Fragment B is 0.5617. In thesame way, the ScorePR values for SurfaceHolder andSurfaceView with Fragment B are all 0.0532.

Table IV. DETAILS OF THE TUTORIAL CORPORA

Table III. RELEVANCE IDENTIFICATION SCHEME
Input: sentence types, ScoreT and ScorePR values between APIs and their
Fragment F, threshold T
Output: the relevance between fragments and APIs;
1 for each (API A in F)
normalize ScoreT(A, F) and ScorePR(A, F);
2
3find all sentences containing A;
4
if (types of these sentences are all marginal sentences)5
A is irrelevant to F;
6else
7
if (normalized ScoreT(A, F)>=T&&normalized ScorePR(A, F)>=T)
8
A is relevant to F;
9
else
10
A is irrelevant to F;

F. Relevance Identification
We design a relevance identification scheme to detect therelevance between fragments and APIs.
In this study, we introduce a threshold T for each tutorial
as one of the inputs of relevance identification scheme,
which is used by all the fragments in the tutorial, to comparewith the two correlation scores to decide the relevance
between fragments and APIs. In the experimental results
section (see VI.A), we will demonstrate how the threshold T
influences the results and how to find a good value of it
automatically.
Table III shows the relevance identification scheme. For
each API in a fragment, we normalize the two correlation
scores between this API and its fragment, namely each scoreis divided by the maximum score of the same type of scoresin this fragment respectively. Then, we check whether all thetypes of sentences containing this API are marginal
sentences. If yes, then we treat that this API only appear inless important sentences, and mark it as irrelevant to thisfragment. If the two normalized correlation scores between
this API and its fragment, namely ScoreT and ScorePR,
exceed the threshold T at the same time, then the API is
identified as relevant to the fragment. Otherwise, the API is
identified as irrelevant to this fragment. With such a scheme,
we can find the relevance between fragments and APIs.
Still taking Fragment B as an example, ScoreT and
ScorePR values for Canvas, Bitmap, SurfaceHolder, andSurfaceView with Fragment B are {1, 0.7862, 0.0139, 0.0023}
and {1, 0.5973, 0.0566, 0.0566} respectively after
normalization. None of these APIs only appear in marginal
sentences, so they need further processing. If the threshold T
is set to 0.29, we can find that both ScoreT and ScorePR of
Canvas and Bitmap with Fragment B are greater than thethreshold T simultaneously. Hence, they are marked as
relevant to Fragment B. Neither ScoreT nor ScorePR of
SurfaceHolder and SurfaceView with Fragment B exceed thethreshold T, so they are marked as irrelevant to Fragment B.
In summary, with FRAPT, Fragment A is marked as a
non-explanatory fragment and filtered out. Canvas andBitmap are relevant to Fragment B, whereas SurfaceHolder
and SurfaceView are not.
G. Relevant Fragment Recommender
In the fragment recommendation phase, Relevant

NonExplanatory
Fragment
explanatory
fragment
with codefragment
JodaTime
36
19
10
21
31
10
16
McGill Math Library 73
Corpus Col. Official 59
31
26
17
[38]
Col. Jenkov 28
34
35
42
Smack
40
42
5
31
Graphics
70
26
12
19
Android Resources 6340
634
Corpus
Data
37
18
7
9
[39]
Text
31
12
12
10
Corpus

Tutorial

API

Fragment
without code
8
25
40
27
16
19
12
16
14

Fragment Recommender recommends relevant fragments
for unfamiliar APIs queried by developers by looking up thedetected relevance between fragments and APIs. For
example, if a developer inputs an unfamiliar API Canvas to
FRAPT. The recommender finds that Fragment B is relevant
to Canvas and recommends Fragment B to the developer.
V.

EXPERIMENTAL SETUP

A. Tutorial Corpora
In the previous studies [3, 7], two publicly open tutorials
are constructed shown in Table IV, which shows the numberof APIs and fragments rather than fragment-API pairs.
McGill corpus consists of five tutorials and Android corpus
is composed of four tutorials. These corpora have been
manually annotated into relevant and irrelevant fragmentswith their contained APIs.
B. Baseline Approaches(1) IR Approach
IR approach uses an information retrieval method to findrelevant fragments for APIs [3]. Cosine similarities with TFIDF term weight are calculated between fragments and API
specifications. A fragment is treated as relevant to an API, if
their cosine similarity is higher than a threshold. The
threshold in each tutorial is defined as follows: first, the K
most similar fragments are achieved by calculating cosine
similarities for each API, where K is the number of relevant
fragments for this API based on the annotation. Then, theaverage of all K-th fragments’ similarities for all the APIs is
used as the threshold.
(2) GMR Approach
GMR approach is the seminal work to find relevant
fragments for APIs [3]. Before extracting features, some texttransformation operations are conducted, e.g., sentence
identification and part-of-speech tagging. Twenty features
are defined to measure linguistic and structuralcharacteristics between fragments and APIs. A MaxEnt
classifier is trained to determine relevant fragments for APIs.
(3) FITSEA Approach
FITSEA approach is the state-of-the-art method to findrelevant fragments for APIs [7]. It introduces some newsources to extend APIs to overcome the information
mismatch between fragments and APIs. Besides, co-Precision =#TP#TP + #FP
#TP

× 100%(4)

Recall = #TP + #FN × 100%
F-Measure =VI.

2 × Precision × Recall
Precision + Recall(5)× 100%(6)

EXPERIMENTAL RESULTS

In this section, we investigate five Research Questions(RQs) to evaluate different aspects of FRAPT.
A. Investigation to RQ1
RQ1: How does the threshold T influence theperformance of FRAPT?
Motivation. The threshold T plays an important role todetermine the relevance between fragments and APIs in the
Relevance Identification component (see IV.F). We try to
explore how the threshold T influences the results anddecide a good value (T0) for T in this RQ.

0.00
0.04
0.08
0.12
0.16
0.20
0.24
0.28
0.32
0.36
0.40
0.44
0.48
0.52
0.56
0.60
0.64
0.68
0.72
0.76
0.80
0.84
0.88
0.92
0.96
1.00

C. Evaluation Methods and Metrics
Supervised approaches can be evaluated by the widelyused Ten-Fold Cross Validation (TFCV) to measure how
accurately an approach performs [34]. We employ TFCV
rather than leave-one-out cross validation (LOOCV) in [3],Since the fundamental unit is the fragment-API pair in
LOOCV, and a fragment containing multiple APIs is
simultaneously put into both the training set and the test set,
which is far away from real applications. In TFCV, we treat
fragments as fundamental units to partition a tutorial into 10
subsets of equal size after identifying APIs and generating
fragment-API pairs. Among the 10 subsets, one subset isused as the test set, and the other 9 subsets are used as thetraining set. After all the subsets are used once as the test set
when cross validation is repeated 10 times, the 10 results areaveraged to evaluate the performance of the approach.
In contrast, unsupervised approaches (FRAPT and IR)
can determine the relevance between fragments and APIs
without the overhead of training. Given a tutorial fragment oran API, unsupervised approaches can discover its relevantAPIs or fragments using their designed frameworks.
In this study, we introduce Precision, Recall, and FMeasure to evaluate the performance of each approach.
These metrics are employed by IR, GMR, and FITSEA in [3,7] as evaluation metrics. Precision measures how accurate
the results of experiments are. Recall indicates the coverageof the results. F-Measure balances Precision and Recall,since there is a tradeoff between them.
There are four kinds of results obtained from theexperiments, namely True Positive (TP), False Negative(FN), True Negative (TN), and False Positive (FP). TPs
indicate the results which correctly predict relevant
fragment-API pairs as relevant. FNs indicate the results
which incorrectly predict relevant fragment-API pairs as
irrelevant. TNs and FPs indicate the results which predict
irrelevant fragment-API pairs as irrelevant and relevant
respectively. Precision, Recall, and F-Measure can becalculated as follows:

The JodaTime tutorial

100%
90%
80%
70%
60%
50%

Precision

100%
90%
80%
70%
60%
50%
40%
30%

Recall

F-Measure

Figure 3. Results of RRAPT on JodaTime tutorial
The Graphics tutorial

0.00
0.04
0.08
0.12
0.16
0.20
0.24
0.28
0.32
0.36
0.40
0.44
0.48
0.52
0.56
0.60
0.64
0.68
0.72
0.76
0.80
0.84
0.88
0.92
0.96
1.00

occurrence APIs are proposed to act as good indicators to
improve the results.

Precision

Recall

F-Measure

Figure 4. Results of FRAPT on Graphics tutorial

Approach. We select one tutorial from the two tutorial
corpora respectively as cases to investigate this RQ, namely
the JodaTime tutorial in McGill corpus and the Graphics
tutorial in Android corpus. We adjust T from 0 to 1 with astep size of 0.01 in the selected tutorials to obtain the results.
Given a tutorial, we try to find a good tutorial-specificvalue threshold T0. By comparing the values of Precision,
Recall, and F-Measure, we can validate whether T0 is close
the optimal value. T0 is derived from the percentage ofsentences containing APIs and their methods in the wholetutorial. This scheme is designed by the observation:
Intuitively, more sentences in a fragment contain APIs andtheir methods, more chances the fragment is relevant toAPIs. Hence, T0 can be calculated as follows:= 1-# sentences in a tutorial containing APIs and their methods# all the sentences in a tutorial(7)

Result. Fig. 3 and Fig. 4 show the Precision, Recall, andF-Measure against T for the two selected tutorials. We cansee that, with the growth of T, the curves of Precision show
upward trends in both tutorials. For example, when T is set to0, FRAPT achieves a Precision of 69.05% in the JodaTime
tutorial. When T comes to 1, the Precision rises to 89.47%.
On the contrary, the curves of Recall show downward trends
in both tutorials. Here, the Recall cannot achieve 100% when
T is set to 0, since some explanatory fragments are filteredout as non-explanatory fragments in the Fragment Filter
component. For example, in the Graphics tutorial, FRAPT
achieves a Recall of 95.12% when T set to is 0, and it drops
to 34.15% when T is set to 1. Compared to Precision and
Recall, the curves of F-Measure are relatively stable.
The black vertical lines in Fig. 3 and Fig. 4 show T0
calculated by the formula (7) in the two tutorials, namely
0.39 and 0.29 respectively. We can see that the values of FMeasure almost approximate the best values using T0 in the
JodaTime tutorial and the Graphics tutorial. Even though
FRAPT does not achieve the best F-Measure, the gap istrivial. For instance, FRAPT achieves the F-Measure of
59.62% in the Graphics tutorial, which is lower than the bestvalue only by 4.21%.

Table V. DETAILED RESULTS OF DIFFERENT APPROACHES
Corpus

Tutorial

McGill Corpus

Android Corpus

JodaTime
Math LibraryCol. Official
Col. Jenkov
Smack
Graphics
Resources
Data
Text

Precision (%)
FRAPT FITSEA GMR
69.00
58.82
85.19
67.89
52.00
84.78
62.03
55.74
62.69
61.19
82.14
90.44
77.94
70.00
83.38
49.21
45.60
50.42
65.22
55.00
75.83
56.52
19.29
71.43
57.58
36.19
66.67

IR
73.00
67.00
30.00
33.00
74.00
35.80
40.32
33.33
37.21

Recall (%)
FRAPT FITSEA GMR
74.17
50.00
76.67
72.70
49.06
73.58
87.50
48.62
31.79
85.17
58.97
97.62
88.33
93.33
94.64
42.52
44.50
75.61
66.17
21.11
66.67
52.00
14.76
55.56
48.33
22.22
76.00

Approach Comparison

100%
80%
60%
40%
20%

Precision

FRAPT

74.23%

FITSEAF-Measure (%)
FRAPT FITSEA GMR
70.24
54.05
80.70
61.53
50.49
78.79
48.10
42.18
72.59
75.23
68.66
85.17
83.90
80.00
85.48
43.73
45.04
59.62
65.93
30.51
66.80
54.17
16.72
62.50
39.56
33.33
65.52

IR
73.00
65.00
94.00
88.00
52.00
67.44
55.56
44.00
57.14

Table VI. RESULTS OF FRAGMENT FILTER
McGill Corpus

Recall

IR
73.00
66.00
45.00
48.00
61.00
46.77
46.73
37.93
45.07

Android Corpus

F-Measure

True Irrelevant

False Irrelevant

True Irrelevant

False Irrelevant

68.46%

63.39%

66

9

27

9

54.74%

52.26%

51.07%

59.08%

46.64%

25.65%

31.40%

58.60%

36.67%

56.04%

44.13%

F-Measure

Precision

86.00%

78.56%

60.86%

73.29%

73.80%

69.79%

GMR

65.13%

56.63%

IR

55.40%

74.40%

McGill Corpus

RecallAndroid Corpus

Figure 5. Average results of each approach

Conclusion. The threshold T may influence theperformance of FRAPT. The tutorial-specific threshold T0 is
close to the optimal value. In the following RQs, we use thethreshold T0 for each tutorial to conduct all the experiments.
B. Investigation to RQ2
RQ2: To what extent is FRAPT superior to the threebaseline approaches over the two tutorial corpora?
Motivation. In this RQ, we want to examine whether our
unsupervised approach FRAPT can be superior to theexisting approaches over the two tutorial corpora.
Approach. FRAPT and the three baseline approaches [3,7] are implemented and tested over the two tutorial corpora
to compare the results.
Results. Table V shows the detailed values of Precision,
Recall, and F-Measure for each approach over the twotutorial corpora. We can see that different approaches
behave differently over the two corpora. FRAPT shows thebest Recall and F-Measure in almost all tutorials. For
example, FRAPT gets the best Precision of 85.19% among
all the approaches in the JodaTime tutorial in McGill corpus.
At the same time, it also achieves the best Recall and FMeasure (76.67% and 80.70% respectively).
The average values of these evaluation metrics arepresented in Fig. 5. We can see that FRAPT achieves thebest results in all the conditions among all the approaches.
For example, the average Precision, Recall, and F-Measureof FRAPT are 74.23%, 86.00%, and 78.56% respectively on
McGill corpus, whereas the other approaches are all below
70% for the F-Measure. FRAPT improves the state-of-theart approach FITSEA by 8.77% and 12.32% on average over
the two corpora in terms of F-Measure. When comparing
different supervised approaches, FITSEA achieves better
results than GMR, e.g., F-Measure of FITSEA is 69.79%
over McGill corpus, whereas GMR only achieves 59.08%.

Table VII. DETAILED RESULTS OF FRAPT AND FRAPT-FILTER
CorpusPrecision (%)
FRAPTFilter

Tutorial

FRAPT

JodaTime
Math Library

McGillCorpusCol. Official
Col. Jenkov
Smack
Graphics
Resources
Data
Text

Android
Corpus

100%

85.19
84.78
62.03
61.19
77.94
49.21
65.22
71.43
57.58

60.00
65.63
38.30
37.84
75.00
43.24
61.82
51.52
42.55

Recall (%)
FRAPTFilter

FRAPT

76.67
73.58
87.50
97.62
94.64
75.61
66.67
55.56
76.00

80.00
79.25
96.43
100.00
96.43
78.05
75.56
62.96
80.00

F-Measure (%)
FRAPTFilter

FRAPT

68.57
71.79
54.82
54.90
84.38
55.65
68.00
56.67
55.56

80.70
78.79
72.59
75.23
85.48
59.62
65.93
62.50
65.52

Results of FRAPT and FRAPT-Filter

80%
60%
40%

Precision

Recall

F-Measure

Precision

McGill Corpus

Recall

F-Measure

Android Corpus

FRAPT

74.23%

86.00%

78.56%

60.86%

68.46%

63.39%FRAPT-Filter

55.35%

90.42%

66.89%

49.78%

74.14%

58.97%

Figure 6. Average results of FRAPT and FRAPT-Filter

Conclusion. As an unsupervised approach, FRAPT canachieve better results than the state-of-the-art supervised
approach verified by TFCV. Considering the advantages of
unsupervised approaches, it is a better choice to use FRAPT
for discovering relevant tutorial fragments for APIs.
C. Investigation to RQ3
RQ3: What is the impact of Fragment Filter on detecting
non-explanatory fragments and improving results of FRAPT?
Motivation. We build a Fragment Filter to detect andfilter out non-explanatory fragments (see IV.C) to address
the non-explanatory fragment identification challenge. We
try to explore its impact on FRAPT in this RQ.
Approach. There are two types of results, namely true
non-explanatories and false non-explanatories. True nonexplanatories are the results in which non-explanatory

Table VIII. DETAILED RESULTS OF FRAPT AND FRAPT-PRONSRES
CorpusPrecision (%)
Recall (%)
F-Measure (%)
FRAPTFRAPTFRAPTFRAPT
FRAPT
FRAPT
PronsRes
PronsRes
PronsRes

Tutorial
JodaTime

McGillCorpusMath LibraryCol. Official
Col. Jenkov
Smack
Graphics
Resources
Date
Text

Android
Corpus

100%

85.19
84.78
62.03
61.19
77.94
49.21
65.22
71.43
57.58

80.00
79.31
56.18
50.00
71.62
53.45
65.12
71.43
55.17

76.67
73.58
87.50
97.62
94.64
75.61
66.67
55.56
76.00

93.33
86.79
89.29
97.62
94.64
75.61
62.22
55.56
64.00

80.70
78.79
72.59
75.23
85.48
59.62
65.93
62.50
65.52

86.15
82.88
68.97
66.13
81.54
62.63
63.64
62.50
59.26

60%
Precision

Recall

F-Measure

Precision

McGill Corpus

RecallCorpus

Tutorial
JodaTime

Precision (%)
FRAPTRecall (%)

F-Measure (%)

FRAPT- FRAPTFRAPT- FRAPTFRAPT- FRAPTFRAPT
FRAPT
Topic
PR
Topic
PR
Topic
PR

85.19 68.29 67.50 76.67 93.33 90.00 80.70 78.87 77.14

Math Library 84.78 82.35 71.64 73.58 79.25 90.57 78.79 80.77 80.00

McGillCol. Official 62.03 56.18 52.63 87.50 89.29 89.29 72.59 68.97 66.23
CorpusCol. Jenkov 61.19 55.41 50.00 97.62 97.62 97.62 75.23 70.69 66.13
Smack
Graphics
Android Resources
CorpusData
Text

77.94 69.23 68.35 94.64 96.43 96.43 85.48 80.60 80.00
49.21
65.22
71.43
57.58

42.35
60.38
48.89
55.88

43.33
49.25
51.06
50.00

75.61
66.67
55.56
76.00

87.80
71.11
81.48
76.00

95.12
73.33
88.89
88.00

59.62
65.93
62.50
65.52

57.14
65.31
61.11
64.41

59.54
58.93
64.86
63.77

Results of FRAPT and variants without topic and PR
100%
80%
60%
40%
Precision
Recall
F-Measure
Precision
Recall
F-Measure

Results of FRAPT and FRAPT-PronsRes

80%
40%

Table IX. DETAILED RESULTS OF FRAPT AND ITS VARIANTS

F-Measure

Android Corpus

FRAPT

74.23%

86.00%

78.56%

60.86%

68.46%

63.39%FRAPT-PronsRes

67.42%

92.33%

77.13%

61.29%

64.35%

62.01%

Figure 7. Average results of FRAPT and FRAPT-PronsRes

fragments are correctly marked as non-explanatory, whilefalse non-explanatories are the results in which explanatory
fragments are incorrectly marked as non-explanatory. After
applying Fragment Filter, we count the number of true nonexplanatories and false non-explanatories to show theeffectiveness of Fragment Filter.
We also compare FRAPT against a variant of FRAPT,
namely FRAPT-Filter which removes Fragment Filter andkeeps the other components the same. By comparing FRAPT
against FRAPT-Filter, the impact of Fragment Filter can befurther shown.
Results. As shown in Table VI, Fragment Filter correctly
marks 66 and 27 true non-explanatories for McGill corpus
and Android corpus and incorrectly marks 9 false nonexplanatories respectively. Fragment Filter can detect a great
number of non-explanatory fragments accurately and only
falsely filter out a fraction of explanatory fragments.
Table VII and Fig. 6 show the detailed and average
results of FRAPT against FRAPT-Filter. We can see from
Table VII that Fragment Filter can largely improve Precision
and F-Measure. For instance, FRAPT obtains F-Measure of
80.70% in the JodaTime tutorial, whereas FRAPT-Filter only
achieves 68.57%. As shown in Fig. 6, introducing Fragment
Filter will effectively improve Precision and F-Measure. For
example, FRAPT achieves an average F-Measure of 78.56%
on McGill corpus, whereas FRAPT-Filter only achieves
66.89% in the same situation. We can also find similar
phenomenon on Android corpus. FRAPT-Filter can achieve
better Recall on average, whereas it shows far worse
Precision and F-Measure. Overall, FRAPT balances
Precision and Recall and shows better results.
Conclusion. Fragment Filter can detect a great number of
non-explanatory fragments at the cost of a small fraction of
explanatory fragments. It can balance Precision and Recall
effectively, thus improve the results of FRAPT.

McGill Corpus

Android Corpus

FRAPT

74.23%

86.00%

78.56%

60.86%

68.46%

63.39%FRAPT-Topic

66.29%

91.18%

75.98%

51.88%

79.10%

61.99%FRAPT-PR62.03%

92.78%

73.90%

48.41%

86.34%

61.78%

Figure 8. Average results of FRAPT and its variants

D. Investigation to RQ4
RQ4: How does pronoun and variable resolution impact
on the performance of FRAPT?
Motivation. We perform pronoun and variable resolution
for each fragment in Fragment Parser to address the pronoun
and variable resolution challenge. To test whether resolution
is effective, we set up this RQ.
Approach. We define a variant of FRAPT, namely
FRAPT-PronsRes which deletes the subcomponent ofpronoun and variable resolution from FRAPT. We compare
the results of FRAPT against FRAPT-PronsRes to show the
impact of pronoun and variable resolution in this RQ.
Results. Table VIII and Fig. 7 show the detailed results
and average results of FRAPT and FRAPT-PronsRes
respectively. We can see from Table VIII that applying
pronoun and variable resolution can improve the results. For
example, FRAPT improves F-Measure by 6.26% in the Text
tutorial of Android corpus. We can see from Fig. 7 that the
average F-Measures of FRAPT-PronsRes over the twotutorial corpora are 77.13% and 62.01% and FRAPT
improves it by 1.43% and 1.38% respectively.
We find that using pronoun and variable resolution
achieves different improvements in distinct tutorial corpora.
There are two potential reasons leading to this phenomenon.
First, different tutorials may contain different numbers ofpronouns and variables, hence different improvements areachieved. Second, pronoun and variable resolution will
enhance the influence of relevant APIs. At the same time, theinfluence of irrelevant APIs may also be strengthened.
Conclusion. Pronoun and variable resolution is aneffective text transformation operation to improve the results.
E. Investigation to RQ5
RQ5: Does FRAPT achieve better results by aggregating
topic model and PageRank algorithm than by any of themalone?

Motivation. FRAPT finds relevant fragments for APIs
by aggregating both topic model and PageRank algorithm.
To explore whether leveraging both of them can improve theresults of FRAPT than any of them alone, we set up this RQ.
Approach. In the same way as the previous RQs, we set
up two variants of FRAPT. The first one is FRAPT-Topic
which only considers the correlation scores from PageRankalgorithm. The second variant is FRAPT-PR which only
takes the correlation scores from topic model into account.
Results. Table IX shows the values of Precision, Recall,and F-Measure over the two tutorial corpora. We can see
from the table that, in almost all the situations, FRAPT
achieves the best results compared to its variants in terms ofPrecision and F-Measure. Taking the Smack tutorial in
McGill corpus as an example, the Precision is 77.94% in
FRAPT, whereas FRAPT-Topic and FRAPT-PR achieve
69.23% and 68.35% respectively. When considering FMeasure, FRAPT outperforms FRAPT-Topic and FRAPTPR by 4.88% and 5.48% respectively. When taking Recall
into account, even though FRAPT shows a slight decline, it
can better balance Precision and Recall to achieve significant
improvement. From Fig. 8 which shows the average values,
we can see that FRAPT shows its advantage.
The reason why aggregating topic model and PageRankalgorithm could improve the results may be that, topic model
can find semantic relationships between fragments and APIs,
while PageRank algorithm tries to find lexical and structuralconnections between them. By aggregating both topic modeland PageRank algorithm, FRAPT could better capture therelevance between fragments and APIs.
Conclusion. By aggregating both topic model and
PageRank algorithm, FRAPT can better detect relevant
fragments for APIs.
VII. THREATS TO VALIDITY
A. Threats to Internal Validity
Internal validity is the degree of our approach to link
independent variables to dependent variables. In theexperiments, we have already investigated the influences ofseveral important components, including Fragment Filter andpronoun and variable resolution, etc. From the results of theRQs, we can find that they are effective components. Besides,
topic model and PageRank algorithm have been applied tomany areas, and their stability and scalability have been
proved. As a result, this threat has been taken into
consideration and minimized as many as possible in FRAPT.
B. Threats to External Validity
External validity is the ability to which the conclusion of
the experiments can be generalized to other corpora andresearch settings. In this study, we introduce all the tutorial
corpora we can obtain. These tutorials are related to Javaand Android APIs, and cover many different topics withdifferent sizes and origins [35]. As a result, they are
representative tutorials for research study. It is unclear how
our approach performs when applying to other tutorials. In
the future, we plan to introduce more tutorials to verify
FRAPT.

VIII. RELATED WORK
There are a lot of API documentations helping
developers to use APIs, e.g., API specifications, APItutorials, forums, and blogs [17-19, 25]. A series of works
exist in the literature related to API documentations [26-33].
We mainly introduce three research topics, i.e. content
classification, information enhancement, and error detection.
Content classification aims to analyze the content of API
documentations to improve the efficiency of consulting API
documentations. Maalej and Robillard [20] developed ataxonomy of knowledge types to study the nature ofknowledge in API reference documentations. Monperrus etal. [21] conducted an empirical study on the directives, and23 kinds of directives were developed and discussed. Dekel
et al. [22] developed and released an Eclipse plugin namedeMoose to show the associated directives for methods.
Information enhancement augments API documentations
by some knowledge units. Treude and Robillard [23]proposed SISE to augment API documentations with insightsentences from Stack Overflow. Kim et al. [24] proposed an
approach to enrich API documentations with code examples.
The results of user study showed that the enriched API
documentations can improve the productivity of developers.
Error detection attempts to find errors in API
documentations. Zhong and Su [19] formulated a set ofinconsistencies and combined natural language processing aswell as code analysis techniques to find these inconsistencies,
and more than 1000 errors were found and reported.
As typical API documentations, we focus on API
tutorials in this study. Different from the above research
topics, we attempt to break up API tutorials into fragments
and find relevant fragments for APIs.
IX.

CONCLUSION AND FUTURE WORK

It remains a challenging issue to discover relevant
fragments for APIs. In this study, we propose an
unsupervised approach, namely FRAPT. Combined with two
fragment examples, we demonstrate how each component
works in FRAPT. We compare FRAPT against three existing
approaches over two publicly open tutorial corpora. The
results show that FRAPT achieves better results than thestate-of-the-art approach. Besides, some key components in
FRATP are also evaluated to show their effectiveness.
For future works, we try to improve FRAPT in several
directions. First, we plan to address the threats to external
validity by introducing more tutorial corpora to validate
FRAPT. Second, we try to adapt FRAPT to discover relevant
fragments for different levels of APIs, e.g., method level.
Third, we try to develop an Eclipse plugin encapsulating
FRAPT to facilitate learning and using APIs for developers.
ACKNOWLEDGMENT
We thank the reviewers for their comments. This work is
supported by the New Century Excellent Talents in
University under Grant NCET-13-0073, the National Natural
Science Foundation of China under Grants 61370144,
61403057, and 61602258, the Fundamental Research Funds
for the Central Universities under Grant DUT14YQ203.

REFERENCES[1]

M. Morisio, M. Ezran, and C. Tully, “Success and Failure Factors in
Software Reuse,” IEEE Transactions on Software Engineering, vol.
28, pp. 340-357, April 2002.
[2] B. Dagenais and M. P. Robillard, “Recovering Traceability Links
between an API and its Learning Resources,” In Proceedings of the34th International Conference on Software Engineering (ICSE 12),2012, pp. 47-57.
[3] G. Petrosyan, M. P. Robillard, and R. de. Mori, “Discovering
Information Explaining API Types using Text Classification,” In
Proceedings of the 37th International Conference on Software
Engineering (ICSE 15), 2015, pp. 869-879.
[4] M. P. Robillard, “What Makes APIs Hard to Learn? Answers from
Developers,” IEEE software, vol. 26, pp. 27-34, November 2009.
[5] M. P. Robillard and R. Deline, “A Field Study of API Learning
Obstacles,” Empirical Software Engineering, vol. 16, pp. 703-732,
December 2011.
[6] L. Ponzanelli, G. Bavota, A. Mocci, et al. “Too Long; Didn’t Watch!:
Extracting Relevant Fragments from Software Development Video
Tutorials,” In Proceedings of the 38th International Conference on
Software Engineering (ICSE 16). 2016, pp. 261-272.
[7] H. Jiang, J. X. Zhang, X. C. Li, et al. “A More Accurate Model for
Finding Tutorial Segments Explaining APIs,” In Proceedings of the
23rd IEEE International Conference on Software Analysis, Evolution,
and Reengineering (SANER 16), 2016, pp. 157-167.
[8] E. Alpaydin, Introduction to Machine Learning (Adaptive
Computation and Machine Learning). The MIT Press, 2004.
[9] R. O. Duda, P. E. Hart, and D. G. Stork, Unsupervised Learning andClustering. Pattern classification (2nd). Wiley, 2001.
[10] V. Stoyanov, C. Cardie, N. Gilbert, E. Riloff, D. Buttler, and D.
Hysom, “Coreference Resolution with Reconcile,” In Proceedings of
the Conference of the 48th Annual Meeting of the Association for
Computational Linguistics (ACL 10), 2010, Short Paper.
[11] Alias-i. 2008. LingPipe 4.1.0. http://alias-i.com/lingpipe.
[12] A. Panichella, B. Dit, R. Oliveto, M. D. Penta, D. Poshynanyk and A.
D. Lucia, “How to Effectively Use Topic Models for Software
Engineering Tasks? An Approach Based on Genetic Algorithms,” In
Proceedings of the 35th International Conference on Software
Engineering (ICSE 13). 2013, pp. 522-531.
[13] S. W. Thomas, Mining Software Repositories with Topic Models.
School of Computing. Queen’s University, Canada. 2012.
[14] Stanford
Topic
Modeling
Toolbox.
http://nlp.stanford.edu/software/tmt/tmt-0.2/.
[15] A. Alon and M. Tennenholtz, “Ranking Systems: The PageRank
Axioms,” In Proceedings of the 6th ACM conference on Electroniccommerce, 2005, pp. 1-8.
[16] A. N. Langville and C. D. Meyer, Google’s PageRank and Beyond:
The Science of Search Engine Rankings. Princeton University Press.
ISBN 0-691-12202-4. 2006.
[17] L. Shi, H. Zhong, T. Xie, et al. “An Empirical Study on Evolution of
API Documentation,” In Proceedings of the International Conference
on Fundamental Approaches to Software Engineering (FASE 11),2011, pp. 416-431.
[18] B. Dagenais and M. P. Robillard, “Using Traceability Links to
Recommend Adaptive Changes for Documentation Evolution,” IEEE
Transactions on Software Engineering, vol. 40, pp. 1126-1146,
November 2014.
[19] H. Zhong and Z. D. Su, “Detecting API Documentation Errors,” In
Proceedings of the 2013 ACM SIGPLAN international conference on
Object oriented programming systems languages & applications(OOPSLA 13), 2013, pp. 803-816.
[20] W. Maalej and M. P. Robillard, “Patterns of Knowledge in API
Reference Documentation,” IEEE Transactions on Software
Engineering, vol. 39, pp. 1264-1283, September 2013.

[21] M. Monperrus, M. Eichberg, E. Tekes, et al. “What Should
Developers Be Aware Of? An Empirical Study on the Directives of
API Documentation,” Empirical Software Engineering, vol. 17, pp.
703-737, December 2012.
[22] U. Dekel and J. D. Herbsleb, “Improving API Documentation
Usability with Knowledge Pushing,” In Proceedings of the 31st
International Conference on Software Engineering (ICSE 09), 2009,
pp. 320-330.
[23] C. Treude and M. P. Robillard, “Augmenting API Documentation
with Insights from Stack Overflow,” In Proceedings of the 38th
International Conference on Software Engineering (ICSE 16), 2016,
pp. 392-403.
[24] J. Kim, S. Lee, S. W. Hwang, et al. “Enriching Documents with
Examples: A Corpus Mining Approach,” ACM Transactions on
Information Systems (TOIS), vol. 31, January 2013.
[25] S. Subramanian, L. Inozemtseva, and R. Holmes, “Live API
Documentation,” In Proceedings of the 36th International Conference
on Software Engineering (ICSE 14), 2014, pp. 643-652.
[26] H. Zhong, L. Zhang, T. Xie, and H. Mei, “Inferring Resource
Specifications from Natural Language API Documentation,” Inproceedings of The 2009 International Conference on AutomatedSoftware Engineering (ASE 09), 2009, pp. 307-318.
[27] D. Kramer, “API Documentation from Source Code Comments: A
Case Study of Javadoc,” In Proceedings of the 17th Annual
International Conference on Computer Documentation (SIGDOC 99),
1999, pp. 147-153.
[28] M. P. Robillard and Y. B. Chhetri, “Recommending Reference API
Documentation,” Journal of Empirical Software Engineering, vol. 20,
pp. 1558-1586, December 2015.
[29] D. Hoffman and P. Strooper, “API Documentation with Executable
Examples”, Journal of Systems and Software, vol. 66, pp. 143-156,
May 2003.
[30] Y. Wu, L. W. Mar, and H. C. Jiau, “CoDocent: Support API Usage
with Code Example and API Documentation,” In Proceedings of 5thInternational Conference on Software Engineering Advances (ICSEA10), 2010, pp. 135-140.
[31] E. D. Ekoko and M. P. Robillard, “Asking and Answering Questions
about Unfamiliar APIs: An Exploratory Study,” In Proceedings of the34th International Conference on Software Engineering (ICSE 12),2012, pp. 266-276.
[32] C. Chen and K. Zhang, “Who Asked What: Integrating Crowdsourced
FAQs into API Documentation,” In Companion Proceedings of the36th International Conference on Software Engineering (ICSE 14), pp.
456-459.
[33] G. Uddin, B. Dagenais, and M. P. Robillard, “Analyzing Temporal
API Usage Patterns,” In Proceedings of International Conference on
Automated Software Engineering (ASE 11), 2011, pp. 456-459.
[34] P. C. Rigby and M. P. Robillard, “Discovering Essential Code
Elements in Informal Documentation,” In Proceedings of the 35th
International Conference on Software Engineering (ICSE 13), 2013,
pp. 832-841.
[35] L. Mastrangelo, L. Ponzanelli, A. Mocci, M. Lanza, M. Hauswirth,and N. Nystrom, “Use at Your Own Risk: The Java Unsafe API in theWild,” In Proceedings of the 30th ACM SIGPLAN Conference on
Object-Oriented
Programming,
Systems,
Languages,
andApplications (OOPSLA 15), 2015, pp. 695-710.
[36] Powers and M. W. David, “The Problem with Kappa,” In Proceedingsof Conference of the European Chapter of the Association for
Computational Linguistics (EACL 12), 2012.
[37] http://docs.oracle.com/javase/tutorial/essential/index.html.
[38] http://cs.mcgill.ca/~swevo/icse2015/.
[39] http://oscar-lab.org/paper/API/.
[40] http://joda-time.sourceforge.net/userguide.html.
[41] http://developer.android.com/guide/topics/graphics/index.html.