Javawock: A Java Class Recommender System Based on Collaborative Filtering
Masateru Tsunoda
Takeshi Kakimoto
Naoki Ohsugi
Akito Monden
Ken-ichi Matsumoto
Graduate School of Information Science,
Nara Institute of Science and Technology
Kansai Science City, 630-0192 Japan{masate-t, takesi-k, naoki-o, akito-m, matumoto}@is.naist.jp

KEYWORDS - information retrieval, cosine similarity,
software component, recommender system, J2SE

1.

INTRODUCTION

Today’s software development platforms provide various types of library components to satisfy varying developers’ demands and needs. Such platforms enable developers to build software that has rich features in shorter
development periods and lower costs [9]. For instance,
Java 2 SDK, Standard Edition (J2SE) Version 1.4.1_02
provides 5568 classes and interfaces as library components.
In spite of a large library provided, most developers
use only a small fraction of the library. For instance, Fig.
1 shows the number of library classes actually used in four
famous open source products. Each product used 224 library classes on average. Since this is only 4% of whole
J2SE library classes, there may be unaware library classes
that can improve developer’s productivity or software
quality. However, it is quite difficult for developers to finduseful library classes just because the library itself is toolarge. We need a system that makes it easy to find usefullibrary class files in each development context.
This paper proposes Javawock, a Java class recom-

mender system based on collaborative filtering (CF). CF is
considered a powerful information filtering method, andhas been used in recommender systems that estimate
end-users’ preferable items (books, movies, tunes, etc).
Typically such system determines items to be recommended in the following way. First, it collects ratings ofitems from many users. A set of ratings of each user iscalled a preference. Next, when a target user was specified,
the system chooses similar users in terms of their preferences. Finally, it determines recommended items by using
preferences of similar users. Usually, items that had highratings in the similar users’ preferences are recommended
if the items are not used (rated) by the target user.
Javawock uses the idea of CF by replacing users withJava programs and items with Java library class files.
Then, a set of library class files used in each program isregarded as a preference of the program. Javawock makesa recommendation of library class files as follows. First, it
collects preferences of programs by investigating usedlibrary class files in each program; then, when a target
program was given, it chooses similar programs in termsof their preferences; finally, it recommends library classfiles by preferences of similar programs.
In what follows, Section 2 introduces related works.
Section 3 explains the detail of CF. Section 4 explains
recommendation procedure and algorithms of Javawock.
Section 5 reports an experiment to evaluate the recommendation accuracy of Javawock. In the end, Section 6
concludes the paper with a summary and some future topics.
6000

5568 (100%)

5000
4000
# of classes

ABSTRACT - Many software development platforms
provide a large number of library components to make iteasy to build high quality software. On the other hand, it
became more and more difficult for developers to finduseful components in each development context because
the amount of components provided became too large
today. This paper proposes a recommender system thatprovides useful Java components (library class files) to adeveloper based on collaborative filtering (CF). When adeveloper gives an unfinished Java program to the system,
it investigates Java library class files used in the given
program and finds Java programs that are similar to thegiven program from a program repository. Then, the system recommends to the developer Java library class files
that were used in the similar programs but were not usedin the developer’s program. An experimental evaluation
showed that the recommendation accuracy of the proposedsystem was much higher than that of a naïve (non-CF)
method in all four evaluation criteria (recall, precision, F1
value, and half-life utility).

3000
2000
1000
0

rt.jar(Library)

Fig. 1.

190(3.41%)

159(2.86%)

110(1.98%)

81(1.45%)Ant

Catalina

JDT-Core

Lucene578(10.38%)Xalan

The number of library classes actuallyused in each product

2.

RELATED WORK

Inoue et al. [4] proposed a Java source code retrieval
system SPARS-J. Although SPARS-J is not a recommender system, it can help developers to find useful Javaclasses based on keyword search. SPARS-J analyzes Javasource programs (*.java files) for a keyword indexing, andit stores both program files and indexed keywords to arepository beforehand. SPARS-J provides developer a
web-based interface including query edit box to enter
keywords. The developer can search program files fromthe repository with keywords related to program’s features,
algorithms or authors. It is intended to be used as“Google” to find Java source files instead of HTML texts.
One drawback of this approach is that it cannot output
relevant programs unless the developer can enter appropriate keywords expressing what the developer wants tosearch. Generally, it is very difficult to choose appropriate
keywords for programs that are unimaginable to the developer. On the other hand, Javawock can recommend
Java class files, regardless of whether or not they are
imaginable to the developer, without giving any keyword
since it requires only an unfinished Java program writtenby the developer as an input.
Ye and Fischer [11] proposed CodeBroker, a non-CF
based recommender system. It recommends a developer
useful Java components (library packages and library classfiles) related to the developer's current task. CodeBroker
automatically extracts keywords from comments in program source codes to capture the development context.
Then, CodeBroker uses these keywords as a query to retrieve components, and recommends them to the developer. Also, CodeBroker recognizes already used components to avoid recommending components already be
known by the developer. The developer does not have toenter any keywords to get recommendation because
CodeBroker automatically gathers keywords from sourcecode. One drawback of this approach is that precision (ratio of appropriately recommended items in all recommended items) greatly depends on quality of comments
written in source code. It requires fully-commented sourcecode stored in the program repository. In our approach ofJavawock, only class files (without source code) are required to be stored in the program repository.
McCarey et al. [5] proposed RASCAL, a CF-based
software component recommender system, which employs
a similar approach to ours. Their system makes a recommendation of Java methods in the following way. First,
the system counts the number of invocations of eachmethod in a Java class file written by the developer, as apreference of the class file. Next, the system finds classfiles similar to the developer's file from a program repository containing other developers’ Java class files. Then,the system recommends methods that were invoked in thesimilar class files but were not invoked in the developer's
class file. There are several differences between Javawockand their system. First, Javawock recommends Java classfiles while their system recommends Java methods. Theirmethod recommendation is useful if there are too manymethods in a class. However, since most of library classes
have not so many methods, we put our priority on classrecommendation with better accuracy. To gain high accuracy, Javawock counts only well-known classes (e.g.
classes in J2SDK and Jakarta project) used in each program to capture the preference of the program, while
RASCAL counts methods of all the classes including locally-developed classes used in each program. This difference affects the similarity computation of programs.
Our approach is based on the idea that a set of well-knownclasses used in a program is considered a native characteristic (birthmark) of the program [10], while local
classes can be easily replaced or changed to other classes.
Second, we employ an item-based CF algorithm as well asa user-based CF algorithm, while RASCAL uses onlyuser-based one. Generally, item-based algorithms aremuch more useful than user-based ones in practical settingwith a large repository. It is because only item-based ones
allow recommender systems to cache the result of similarity computation (For this reason, Amazon.com uses anitem-based algorithm in their book recommender system).
Third, we used more criteria to evaluate the recommendation accuracy.

3.

COLLABORATIVE FILTERING

CF is one of the key techniques for implementing a
recommender system that recommends to a user a set ofcandidate items which may be preferable or useful to theuser. For example, Resnick et al. [6] developed GroupLens system which recommends interesting Usenet articles to users. It draws on a simple idea; people who agreed
in their subjective evaluation of past articles are likely toagree again in the future. Resnick et al. proposed a basic
CF algorithm known as a user-based method.
User-based method makes recommendations with thefollowing procedure:
1. After using items (Usenet articles, books, movies, etc.),users explicitly assign numeric ratings to the items.
2. A recommender system correlates the ratings in order todetermine which user’s ratings are the most similar toother ones.
3. The system predicts ratings of new items for the target
user, based on the ratings of similar users.
4. If these new items seem to be preferred, the system recommends them to the user.
Sarwar et al. [8] proposed another basic CF algorithm
called item-based method. Item-based method can makerecommendation with extremely sparse dataset whose
ratio of rated items to whole items is only 1%. Item-based
method makes recommendations with executing 2’ and 3’
instead of 2 and 3 in the above procedure respectively:
2’. A recommender system correlates the ratings in orderto determine which item’s ratings are most similar toother item’s.
3’. The system predicts ratings of new items for the target
user, based on the ratings of similar items alreadyUnfinished Java Program(Java class file)

Preference of
the Unfinished Program1

2
Web FEP

User

5
Recommendation

CF Engine

4
Recommendation
3
Preferences of
the Other Programs

Javawock

Fig. 2.

PR

The architecture of Javawock

l1

l2

…

lj

…

lb

…

ln

p1

u1,1

u1,2

…

u1,j

…

u1,b

…

u1,n

p2

u2,1

u2,2

…

u2,j

…

u2,b

…

u2,n

…

…

…

pi

ui,1

ui,2

…

…

…

pa

ua,1

ua,2

…

…

…

pm

um,1

um,2

…
…

ui,j

…
…

…
…

ua,j
um,j

…

…
…

…
…

ui,b

…

ua,b

…
…

…
…

um,b

ui,n
ua,n
…

…

um,n

Fig. 3. m×n table used for recommendation
rated by the users.
Intuitively, an idea of the item-based method can be represented as a popular sentence of Amazon.com’s recommendations “Customers who bought this book alsobought ...”.
Javawock assumes each Java program (a class filewritten by a developer) as a user, and assumes each Javalibrary class file used in the program as an item (Note thata class file can become both the user and the item since
the class file itself uses a library class file). Javawock analyzes which Java library class files were used in each developer's class file. Then, Javawock puts high ratings tolibrary classes used in the developer’s class file, and low
rating to library classes unused in it. We implemented
both user- and item-based method as Javawock’s CF algorithms. With the user-based method, Javawock predicts
which library class files will be used by the developer,
based on the similarity of how each class uses libraryclasses. With item-based method, Javawock predicts
which library class files will be used by each developer,
based on the similarity of how each library classes areused by other classes.

4. RECOMMENDING JAVA CLASS FILES
4.1. Architecture of recommender System
Fig. 2 shows the architecture of Javawock. Javawock
consists of a web-based frond-end processing (web FEP),
a collaborative filtering engine (CF engine) and a program
repository (PR). Javawock recommends Java library classfiles for a Java program uploaded by user. The uploaded
Java program is a class file (not a source file), thus, theusers must compile their Java program before uploadingit.
Javawock recommends library class files as follows:
1. User uploads an unfinished Java program (target program) to the web FEP.
2. The web FEP analyzes the target program to get a setof library class files used in the given program (preference). To get the preference, the web FEP uses
Used-Class analysis engine proposed in jbirth [10].
3. The web FEP sends the preference to the CF engine.
4. The CF engine receives the preference and makes arecommendation by using other programs’ prefer-

5.
6.

ences stored in the PR.
The CF engine returns the recommendation result to
the web FEP.
The web FEP shows a user the recommendation. The
recommendation result consists of library class names,
recommendation scores, abstracts of Java API documents and links (URLs) to them.

Fig. 4 shows an input screen that uploads class file,and Fig. 5 shows an output screen of the recommendation
result. Javawock has a Google like interface. Each recommended library class is linked to the Java API document.

4.2.

Recommendation methods

Javawock provides three types of recommendations.
The first one employs a user-based CF method. With thismethod, Javawock makes recommendation based on thesimilarity between programs. The similar programs use
similar library class files, compared with the target program. The user-based method proceeds as follows:
1. Javawock chooses several similar programs from PR.
2. Javawock computes a recommendation score of eachlibrary class file whom similar programs use and thetarget program does not use.
3. Javawock shows recommended library class files
ranked by their score.
The second method employs an item-based CF method.
Javawock makes recommendation based on the similarlity
between library class files (similarity of how they areused). The class file l0 and l1 are considered similar eachother if l0 and l1 are used in the same set of programs. The
item-based method proceeds as follows:
1. Javawock chooses several similar library class files
for each library class file.
2. Javawock computes a recommendation score of eachlibrary class file li if the target program does not use it
but uses a library class file lk that is similar to li.
3. Javawock shows recommended library class files
ranked by their score.
The third method simply outputs Java class files thatare similar to the given program based on the user-based

Fig. 4.

Fig. 5.

The input screen of Javawock

similarity computation algorithm. (Note that we haven’t
experimentally evaluated this method in this paper).

4.3.

Collaborative Filtering Algorithm for the
Recommendation

The CF engine computes the similarity and the recommendation score to make a recommendation. Whenmaking a recommendation, CF engine uses the program
repository in form of m×n matrix as shown in where pi∈{p1, p2, ..., pm} denotes i-th program, lj∈{l1, l2, ..., ln}denotes j-th library class file, and ui,j∈{u1,1, u1,2, ..., um,n}denotes status of whether library class file lj is used or notby program pi. If program pi uses library class file lj, thevalue of ui,j (library class status) is set to 1, and if program
pi does not use library class file lj, the value of ui,j is set to
0. So there is no missing value in the matrix.
Our CF engine uses the cosine similarity algorithm tocompute similarity. Similarity is computed to choose
similar programs or similar library class files. Althoughvarious similarity computation algorithms are proposed
for CF [1][2][6][8], we selected the cosine similarity algorithm because it showed the highest accuracy in our pilot
experiment. This algorithm was originally proposed toevaluate the similarity between two documents in the fieldof information retrieval. The similarity is often evaluatedby treating each document as a vector of word frequencies
and computing the cosine of the angle formed by the two
frequency vectors [7].
On the user-based method, the similarity, sim(pa, pi)
between the target program pa and other program pi isformally defined as the following (1). In this equation,
programs, library class files and library class statuses areused instead of documents, words and word frequencies.
The value range of sim(pa, pi) is [0, 1].

sim( p a , p i ) =

∑u

a, j

∑ (u a , j ) 2

× ui, j(1)

∑ (ui , j ) 2

In the item-based method, similarity sim(lb, lj) between
the library class file lb and other library class file lj is formally defined as (2).

sim ( l b , l j ) =The output screen of Javawock

∑u

i ,b

× ui, j(2)

∑ (u ) ∑ (u )
2

i ,b

2

i, j

Our CF engine uses the weighed sum algorithm [8] tocompute the recommendation score. The recommendation
score is a predicted value of ua,j owned by program pa.
The CF engine predicts ua,j of each library class file which
is not used by the target program. Although various recommendation score computation algorithms are proposed
for CF [1][2][6][8], we selected the weighed sum algorithm because it showed the highest accuracy in our pilot
experiment.
When computing a score, the CF engine does not use
all similar programs or all similar library class files, butuses k similar programs or k similar library class files. k iscalled neighborhood size.
In the user-based method, the recommendation score
Ra,b, which is a predicted value of ua,b owned by programpa is formally defined as (3). The recommendation score iscomputed with the weighed average of ui,b owned bysimilar programs pi. Each weight is similarity between thetarget program pa and each program pi. k-nearestPrograms means a set of k similar programs.

∑ sim ( p , p ) × u∑ sim ( p , p )a

R a ,b =K

K ,b

K ∈k − nearestPro grams

a

(3)

K

K ∈k − nearestPro grams

In the item-based method, the recommendation score
Ra,b, which is a predicted value of ua,b owned by programpa is formally defined as (4). The recommendation score iscomputed with the weighed average of ua,j owned bysimilar library class files lj. Each weight is similarity between library class file lb and other library class lj.
k-nearestLibraryClassFiles means a set of k similar library class files.

∑ sim(l , l ) × u

R a ,b =b K
K ∈k − nearestLibraryClassF iles

∑ sim(l , l )b KK ∈k − nearestLibraryClassFiles

a,K

(4)

F1 Value

60%
40%
20%
0%
10%

30%

50%

70%

Half-Life Utility

User-based
Item-based
Average-based

80%

80%

User-Based60%

Average-based

Item-Based

40%
20%
0%
10%

90%

30%

Known Components

Fig. 6.

F1 value of each method

Fig. 7.

5. EXPERIMENTAL EVALUATION
5.1. Dataset
For an experimental dataset, we extracted classes fromrt.jar, class library of J2SE (Java 2 Platform Standard Edition) Software Development Kit. We selected 371 commonly used classes as programs P={p1, p2, ..., pm,} and331 classes as library class files L={l1, l2, ..., ln}, which isa subset of P. The number of library class files n is smaller
than that of programs m because we excluded 40 libraryclass files (from L) that were not used by any program in
P, Consequently, we made 371×331 size dataset from
them.

5.2.

Experimental Procedure

In the experiment, we evaluated both the user-based
method and the item-based method. The experiment proceeds as follows (leave-one-out cross-validation):
1. i-th program pi is regarded as the target program, and
it is removed from the dataset.
2. ui,j is regarded as unknown, and Javawock computes
Ri,j, a recommendation score for library class file lj(i.e. Javawock predicts the value of ui,j).
3. Repeat Step 2 for all j.
4. Repeat Step 1, 2, 3 for all i.
In order to virtually produce an unfinished program ofPi, we also used a criterion q, denoted as Known Component in Fig 4,..,8, which is a percentage of library classfiles regarded to be already used in Pi to the total libraryclass that will be used when Pi is finished. We varied q

R a ,b =

5.3.

Nl
Nc

(5)

Evaluation Criteria

We used four criteria (recall, precision, F1-value andhalf-life utility) to evaluate recommendation accuracy ofthe proposed methods. These are often used to evaluate
accuracy of CF based system [3]. The higher these values
are, the more accurate evaluated method is.
Precision is a ratio of appropriately recommended library class files to entire recommended library class files,
formally defined as (6), where Nr is the number of entire
recommended library class files, and Na is the number ofappropriately recommended library class files. That is, Nr

User-Based
Item-Based
Average-based

80%

Average-based

40%
20%

Precision

Recall90%

Half-life utility of each method

Item-Based

60%

70%

from 10% to 100% at 10% intervals by step-by-step adding actually used library class files in Pi. Note that at least
one library class file is remain unknown even if q=100%(see Step 2).
We conducted pre-examination to define theneighborhood size k used in (3) and (4), and defined k to
be 3 for both (3) and (4) since recommendation accuracy
became the highest in this case.
We also computed recommendation scores using anaïve (non-CF) method called average-based method tocompare with proposed methods. In the average-based
method, the recommendation score Ra,b, predicted value of
ua,b owned by program pa is formally defined as (5), where
Nc is the number of entire programs, and Nl is the numberof programs that uses library class file lb.

User-Based

80%

50%

Known Components

60%
40%
20%

0%
10%

30%

50%

70%

90%

0%
10%

30%

Known Components

Fig. 8.

Precision of each method

50%

70%

90%

Known Components

Fig. 9.

Recall of each method

User-based
Item-based

80%

Average-based

Precision

60%
40%
20%
0%
30%

50%

70%

90%

RecallFig. 10.

The relation between recall and precision

is the number of scores that satisfy Ri,j=1, and Na is thenumber of scores that satisfy both Ri,j=1 and ui,j (predictedvalues of Ri,j) =1.

Precision =

Na
Nr

(6)

Recall is a ratio of appropriately recommended libraryclass files to entire library class files actually used in the
program pi, formally defined as (7), where Nu is the number of entire library class files actually used in pi, and Na isthe number of appropriately recommended library classfiles. That is, Nu is the number of scores that satisfy ui,j=1,
and Na is the number of scores that satisfy both Ri,j=1 andui,j=1.

Recall =

Na
Nu

(7)

F1 value is a combined criterion of recall and precision, formally defined as (8).

F1 =2 × Precision × Recall
Precision + Recall(8)

Half-life utility (H) is a criterion to evaluate the ranking based on recommendation score, formally defined as(9) and (10), where sa,d is the predicted library class statusof library class file lj recommended to the target program
pa at d-th rank (That is, the value of sa,d is same as the predicted value of ua,j). α is the rank of a recommended library class file whom users will view with a probability of
50%. We defined α to be 10. Hamax is equal to Ha when thesystem makes perfect recommendation.

Ha = ∑dsa,d( d −1) /(α −1)

(9)

∑H
∑H

(10)

2

H = 100 ×

a

a

5.4.

a
max
a

Experimental Result

Experimental result is shown in Fig. 6,…,Fig. 10.
These graphs show that user-based method has good rec-ommendation accuracy. Fig. 6 shows the relation between
the highest F1 value of each method and the percentage ofknown components (library class files considered to bealready used by the developer). We changed threshold tofind the highest F1 value at each percentage of known
components. The line of the average-based method in Fig.
6 is parallel to x-axis because average-based method is
independent of pi to make recommendation.
Fig. 6 shows that the user-based method is the mostaccurate of the three and the average-based method is themost inaccurate. In this figure, when known components≈100%, i.e. programming is almost finished, F1 value ofthe user-based is 66%, that of the item-based is 46%, andthat of the average-based is 18%. When q<20%, theitem-based method is less accurate than the average-based
method. This indicates that the developer needs to build atleast 20% of the target program to get better recommendation when he/she wants to use the item-based method. Fig.
7,…, Fig. 9 show other criteria when F1 value is the highest. The trends in these graphs are similar to Fig. 6.
Fig. 10 shows the relation between recall and precisionof each method (when q≈100%). The curve of theuser-based method is always the highest of three methods
and that of the average-based is always the lowest. From
Fig. 6,…, Fig. 10, we conclude that the user-based method
is always more accurate than the average-based method,
and the item-based method is more accurate than the averaged-based method if q≥30%.

6.

CONCLUSION

In this paper, we proposed Javawock, a Java class recommender system, based on CF. Javawock employs bothuser-based and item-based method to make recommendations. An experimental evaluation showed that theuser-based method is always more accurate than the naïve
(non-CF) average-based method, and the item-based
method is more accurate than the averaged-based method
if the percentage of known components≥30%.
The limitation of our experiment is that we used only
one dataset for evaluation. We will conduct further experiment using other datasets to extensively evaluate theproposed method.

ACKNOWLEDGEMENTS
This work is supported by the EASE (Empirical Approachto Software Engineering) project of the Comprehensive
Development of e-Society Foundation Software programof the Ministry of Education, Culture, Sports, Science andTechnology of Japan, and Grant 15103 of the Open Competition for the Development of Innovative Technology
program.

REFERENCES
1.

J. Breese, D. Heckerman, and C. Kadie, “Empirical
Analysis of Predictive Algorithms for Collaborative
Filtering,” in Proc. Conf. on Uncertainty in Artificial
Intelligence, Madison, WI, pp. 43-52, Jul. 1998.

2.

D. Goldberg, D. Nichols, B.M. Oki, and D. Terry,“Using collaborative filtering to weave an information tapestry,” Communications of the ACM, vol..35,
no.12 pp. 61-70, Dec. 1992.
3. J. Herlocker, J. Konstan, L. Terveen, and J. Riedl,“Evaluating collaborative filtering recommender systems,” ACM Transactions on Information Systems(TOIS), vol.22 , no.1, pp.5-53, 2004.
4. K. Inoue, R. Yokomori, H. Fujiwara, T. Yamamoto,
M. Matsushita, S. Kusumoto, “Component Rank:
Relative Significance Rank for Software Component
Search,” In Proc. International Conference on Software Engineering, pp.14-pp.24, Portland, Oregon,
2003.
5. F. McCarey, M. Ó Cinnéide, and N. Kushmerick,“RASCAL: A Recommender Agent for Software
Components in an Agile Environment,” In Proc. Artificial Intelligence and Cognitive Science Conference,
Castlebar, Ireland, pp.107-pp.116, Sep. 2004.
6. P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom,and J. Riedl, “GroupLens: An Open Architecture forCollaborative Filtering of Netnews,” In Proc. ACM
Conf. on Computer Supported Cooperative Work,
Chapel Hill, NC, pp.175-pp.186, Oct. 1994.
7. G. Salton, and M. McGill, Introduction to Modern
Information Retrieval. New York: McGraw-Hill,
1983.
8. B. Sarwar, G. Karypis, J. Konstan, and J. Riedl,“Item-Based Collaborative Filtering Recommendation Algorithms,” In Proc. International World Wide
Web Conference, Hong Kong, China, pp. 285-295,
May 2001.
9. C. Szyperski, Component Software: Beyond Object-Oriented Programming. New York: Addison-Wesley, 1998.
10. H. Tamada, M. Nakamura, A. Monden, and K. Matsumoto, “Design and evaluation of birthmarks for
detecting theft of Java programs,” In Proc. IASTEDInternational Conference on Software Engineering,
Innsbruck, Austria, pp.569-575, Feb. 2004.
11. Y. Ye, and G. Fischer, “Supporting Reuse by Delivering Task-Relevant and Personalized Information,”
In Proc. International Conference on Software Engineering, Orlando, FL, pp.513-523, May 2002.