2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed
Computing

A Study of Student Experience Metrics for Software Development PBL
Umekawa Kohichi, Hiroshi Igaki, Yoshiki Higo, Shinji Kusumoto
Graduate School of Information Science and Technology
Osaka University
Osaka, Japan
k-umekaw@ist.osaka-u.ac.jp, igaki@ist.osaka-u.ac.jp, higo@ist.osaka-u.ac.jp, kusumoto@ist.osaka-u.ac.jp

effort of students and teachers. Henceforth, in section 2,
we describe the existing research on evaluation methods
for the software development PBL. Section 3 denotes
about our proposed method including several metrics forevaluating students’ experience. Section 4 presents implementation of our system. We write an experiment and a
discussion about our system in section 5 and 6.

Abstract—In recent years, the increased failure originatedin the software defects, in various information systems
causes a serious social problem. In order to build a highquality software, cultivation of ICT(Information and Communication Technology) human resources like a software
engineer is required. A software development PBL(Projectbased Learning) is the educational technique which lets
students acquire knowledge and skill spontaneously throughpractical software development. In PBL, on the other hand, it
is difﬁcult to evaluate not only the quality of the product but
also the quality of the development process in the project.
In this paper, we propose the student evaluation metrics
to assess the development process in PBL. The student
evaluation metrics represent LOC(Lines of Code) and development time for each product developed by a student. By
using online storage, these metrics can be measured andvisualized automatically. We conducted an experiment toevaluate the accuracy of the metrics about development time.
As a result, we conﬁrmed that development time metrics canbe measured with approximately 20% of error.

II. P RELIMINARY
A. Software Development PBL
In this paper, we call the software development PBL a
SDPBL[2]. The following subjects may be given to thegroup of the students who develop a project, in addition
to the feature of common PBL.
T1: Can the product which fulﬁlls given speciﬁcation
be implemented within a certain period deliberately?
T2: Are the tasks appropriately shared within the
project group?
T3: Can the required tasks fully be carried out in thegroup?
The SDPBL is an educational approach which trains
students devising Project Management skills for smoothsoftware development process spontaneously. Therefore,
the student group needs to manage their development process in the viewpoint of time for delivery, task assignmentand software quality. T1 is related to time for delivery.
The student group manages software development process
and must protect the given time for delivery. T2 indicatesthat every student should have the same chance to acquire
experience of software development in the SDPBL. The
student group needs to progress software development
efﬁciently, taking T2 into consideration. T3 is related tosoftware product quality. In order to develop high-quality
software, students must perform not only implementing
but tests and reviews for products.
ITSpiral[3] contains the SDPBL program. In the
SDPBL program, through the experience of group software development, the students learn the difﬁculty ofworking in a group and the importance of all-aroundskills such as communications, schedule management, andproject management. Furthermore, through looking thedevelopment process back, the students obtain the further
understanding of the development process and process
improvement.
Shibaura Institute of Technology performs a class about
PBL which is named advanced information practice 1B[4].

Keywords-metrics; PBL; process improvement; visualization;

I. I NTRODUCTION
In recent years, software problems are increasing, and
it is becoming a serious social problem. It is pointed outthat quality and quantity of talented people for building
good software are insufﬁcient. Training of sophisticated
ICT(Information and Communication Technology) talented people is required in the industrial world. In theuniversity, including department of information, educationand learning technique called PBL(Project-based Learning) based on actual software development are performed
with various forms[1]. The PBL is an educational system
which aims at advanced ICT talented people’s training andprovides an opportunity of autonomous study for students.
Students form a group and tackle several practical tasks.
In this paper, we focus on the software development PBL
in which students develop actual software product. In
such kinds of PBL, students can learn skills of project
management or communication practically through thesoftware development. Usually, multiple students in thesame group cooperate in the software development. Then,
it is difﬁcult for a teacher to evaluate how much theindividual student contributed to the development group.
We propose a method which collects development logs
of each student in the software development PBL withusing an online storage service. Based on the method,
we realized analysis and visualization environment which
evaluate experiences of the individual student with little
978-0-7695-4761-9/12 $26.00 © 2012 IEEE
DOI 10.1109/SNPD.2012.76

465

K2:

Detailed development experience metrics for every student based on the development logs.
K3: Contribution evaluation method by the comparison of development experience metrics in agroup.
We use the online storage service ”Dropbox”[6]. Dropbox
shares data in a user’s HDD between two or more PCs
through the online storage service on the Internet.
We share students’ working directories with an instructor using the online storage. The online storage has
the function of Version Control System, and the storage
records contents of ﬁles whenever a user saves ﬁles. Our
system collects contents of ﬁles recorded by the storage as
the user’s ﬁne-grained development log. Based on the collected development log, our system measures experienceof each student such as development time and lines ofcode. Comparing an individual’s measured quantitative
experience within a group indicates contribution degree toa project for each student. Henceforth, we explain thesekey ideas in detail.

In the class, groups of students develop several kindsof products such as documents and source codes. This
class aims at training of creativity and the problem-solving
capability for advanced ICT talented people.
SDPBL environment differs for every school. Deﬁnitions of the environment which is used in many SDPBLs
are shown below.
P1: Development environment to work on the development project is used by the students.
P2: Development support environment supports thedevelopment process.
P3: Teaching support environment facilitates theanalyses of the development process by instructors.
B. Evaluation for Each Student and Group in SDPBL
Evaluation in SDPBL is performed for each group andeach student. An instructor in SDPBL usually adopts theevaluation approach shown below.
E1: Evaluation by an instructor’s subjectivity
E2: Evaluation by an examination or a report about
SDPBL by each student
E3: Evaluation by a questionnaire by each student
and group
In E1, an instructor observes and estimates the progressof development of each group and student. The instructor
evaluates subjectively, and load of the evaluation is soheavy. In E2, an instructor gives a report or an examination
based on the group development in SDPBL to the students
and evaluates by the result. Although the report and theexamination are essential in the view point of estimationof comprehension about the group development, it isnecessary to examine carefully whether the examinationor the report are in agreement with the contents of the
SDPBL. The E3 contains a group questionnaire and a
student questionnaire. The questionnaires are signiﬁcantin the meaning of grasp of students’ subjective evaluation
qualitatively.
As a result, almost all conventional evaluation approachaims at estimating students and groups qualitatively. The
software management method PSP(Personal Software Process) is one of the method which evaluates a student
quantitatively[5]. In the PSP, individual development data
are recorded based on strictly-deﬁned metrics. The development process based on the PSP enables evaluation
using recorded individual development data. However, thedevelopment data collected by each developer manually
may often be inaccurate. Then, we propose the metrics
which enables evaluation of each student development
experience in the SDPBL.

[{"rev": "40000000d","bytes": 0,"modified": "Wed, 20 Jul 2011 22:41:09 +0000","path": "/hi2","is_dir": false,"mime_type": "application/octet-stream"},{"rev": "10000000d","bytes": 3,"modified": "Wed, 20 Jul 2011 22:40:43 +0000","path": "/hi2","is_dir": false,"mime_type": "application/octet-stream"}]

Figure 1.
service

Version information examples by the API of the storage

A. Collecting Development Log using Online Storage Service
The online storage service ”Dropbox” has the following
features.
F1: The data in a folder speciﬁed by a user are saved
automatically at the storage on the service.
F2: The storage service has a feature of Version
Control System.
F3: The folder speciﬁed by a user is sharable withother users.
F4: APIs for accessing the storage service are exhibited.
By using the online storage service, whenever a user
updates any ﬁle, time stamp, and the contents of ﬁles
are recorded. All history of the recorded ﬁles can beaccessed through the APIs by users. The API of thestorage service returns version information for every ﬁleas shown in the ﬁgure1. In the version information, ”rev”,”bytes”, ”modiﬁed”, ”path”, and ”mime type” indicate aversion number, a size of the ﬁle, an updated time, pathinformation of the ﬁle, and a type of the ﬁle respectively.

III. S TUDENT E XPERIENCE E VALUATION M ETRICS
FOR S OFTWARE D EVELOPMENT PBL
Our proposed student experience evaluation metrics
consists of the following key ideas.
K1: Automatic collection of development logs byusing an online storage service.

466

Figure 2.

Example of the accumulated LOC

Figure 4.

Figure 3.

Example of comparison about aLOC within a groupresult, the accumulated LOC(aLOC) of the ﬁle A to ver3
is add 15 and change 5, or simply 20 aLOC.
The 2nd metrics is an accumulated development timefor every ﬁle. The online storage records contents of theﬁle information on the service, whenever a user saves aﬁle. Then, we measure the accumulated development time
based on the update time of all ﬁles saved by a user.
Step1: Collects history of update time of all ﬁles which
are saved by a certain user.
Step2: Sorts all versions of all the ﬁles in order of
update time.
Step3: Calculates the diff of the update time between
two continuous versions of ﬁles.
Step4: Adds the diff of the update time to the ﬁleupdated subsequently as the accumulated development time of the ﬁle, if the diff of the update
time is below predeﬁned threshold.
The ﬁgure 3 shows an example of the accumulated
development time. In the ﬁgure, all versions of all ﬁles are
sorted in order of update time. The diff of the update time
between ver1 of ﬁle A and ver 1 of ﬁle B is 20 minutes.
Here, suppose that threshold about the diff of update timeis 1hour. The diff of the update time 20 minutes is added
to the accumulated development time of ﬁle B. Similarly,
15 minutes and 28 minutes are added to the accumulated
development time of ﬁle C and ﬁle B respectively. As aresult, the accumulated development time(aDEV) of the
ﬁle A, B, C in this example are calculated as 0 minutes,
48 minutes and 15 minutes respectively.
As the above example shows, the online storage service
cannot record the start time of user’s development. We
have to take into consideration the error about the development start time, when we measure the accumulated
development time.

Example of the accumulated development time

By using such online storage service and APIs, theﬁne-grained development log than the existing version
control system is collectable for each user. In the following
section, we describe our metrics using this development
log.
B. Development Experience Metrics of Each User
Development Experience Metrics contains two kinds ofmetrics. The 1st metrics is an accumulated LOC of a
product implemented by a student until a certain time.
When a student implements a product, addition, deletion,and change to the ﬁle are performed. Here, we deﬁne theaccumulated LOC as the total of the lines of code addedby the student, after the ﬁle is newly created. In addition,
the number of change lines is included in the accumulated
LOC as the number of lines is added after deletion.
In order to measure the accumulated LOC, all theversion information of the ﬁles implemented by each
student is collected through the API of the online storage
service. As shown in the ﬁgure 2, suppose that two or moreversions of a certain ﬁle A exist in the online storage. Our
system acquires all the version information of a certain ﬁlethrough the API and calculates the diff between adjoining
version.
In this example, the diff between ver1 and ver2 of the
ﬁle A is add 5(LOC) and change 2(LOC). Similarly, thediff between ver2 and ver3 is add 10 and change3. As a

C. Evaluating Contribution of Students in Development
Group
We calculated the accumulated LOC and the accumulated development time of each product ﬁle which each467

Figure 6.

Figure 5.

Magnitude of the error rate in application development

Example of comparison about aDEV within a groupstudent has developed on the online storage. Visualizationof comparison about our metrics within a group canmake students recognize the deviation of the work loadof group development. The ﬁgure 4 shows an exampleof the bar graph about aLOC for every member within agroup. This example indicates that the difference of aLOC
between member 1,5 and member 2,3 and 4 is very large.
Considering that the SDPBL is the educational approach,
it is desirable that the experience which students get tobe qualitatively or quantitatively similar among several
students.
Similarly, the ﬁgure 5 shows an example of the pie chartindicating the percentage of the accumulated development
time in the group. Combining Figure 4 and Figure 5 makes
it easy to grasp the situation of SDPBL. For example,
although the aDEV of member 1 is long, there are few
aLOC of the member. The monitoring result may imply
that the member 1 tackles too difﬁcult problem about hisdevelopment.

Figure 7.

Magnitude of the error rate in paper writing

error about the accumulated development time measured
by our system.
In the experiment, 6 students developed products on
the online storage service. We used an application called
WorkTimer to record actual development log, i.e., operation start time, and ﬁnish time for each development task
manually.
Based on the actual development log for about two days
captured by each student with WorkTimer, we analyzed
the error between aDEV and the actual development log.
In addition, students conducted two kinds of developmenttask. One is Java application development and another ispaper writing. Figure 6 and 7 show the relation between
the value of the threshold for calculating the aDEV, andthe magnitude of the error in application development andpaper writing, respectively. In the ﬁgure, a threshold isdisplayed on the horizontal axis, and the magnitude of theerror of aDEV is displayed on the vertical axis.
The result of the ﬁgure 6 shows that the error decreased
to about 20.5 %, when we set up the threshold to 50
minutes in application development. Another result of theﬁgure 7 shows that the error decreased to about 31 %,when we set up the threshold to 80 minutes in application
development. The magnitude of the error varied with thekinds of task. This difference is considered to originate
from the difference of development environment. In thisexperiment, every student developed the Java application

IV. I MPLEMENTATION
We adopted Java SE 6 update 20 and JavaScript as
development languages for our system and developed on
the Eclipse 3.7 and the NetBeans IDE 7.0.11. We used
Dropbox4j[7] to access Dropbox API using OAuth, anddb4o for saving proposed metrics as DBMS. Furthermore,
DWR framework[8] was adopted for integrating Java and
JavaScript to create our Ajax web application. We used
jQuery and jqplot[9] of the JavaScript library for the
purpose of visualizing our metrics.
V. E XPERIMENT FOR ESTIMATING ERRORS OF THE
ACCUMULATED DEVELOPMENT TIME

Although we can collect the update time of each ﬁle, wecannot know a student’s operation start time. Since exact
operation start time cannot be obtained, the threshold or awork environment may affect the magnitude of the error
on the aDEV. Therefore, in this research, we conducted anevaluation experiment for checking the magnitude of the468

on Eclipse IDE, and wrote the paper using a text editor. In
the development on the Eclipse, the students have to savethe source code ﬁles at the time of compile or execution of
the Java application. In the development on the text editor,
the students save ﬁles by their decision. That is, in theapplication development, ﬁles are saved more frequently.

[2] N. Fukuyasu, S. Saiki, Y. Mizutani, H. Igaki, and Y. Manabe, “An adaptive teaching on software development PBL,”
Japanese Society for Information and Systems in Education,
vol. 26, no. 7, pp. 169–176, 3 2012, (in Japanese).
[3] IT Spiral, “IT Specialist Program Initiative for Realitybased Advanced Learning,” http://it-spiral.ist.osakau.ac.jp/index.html.

VI. D ISCUSSION[4] S. Komiya, “Exercise Class for Software Development
technique
based
on
PBL,”
http://www.gracecenter.jp/downloads/itsp/talk7.pdf, 2011, (in Japanese).

In this paper, we proposed two kinds of metrics to measure and evaluate the experience of the students on groupdevelopment in SDPBL. Those metrics can be measured,
merely by setting up the online storage service. In theexperiment, we checked that the accumulated development
time was measurable with about 30 In almost all the
results of the experiment, values smaller than the actual
development time measured by the student manually were
calculated as aDEV. We consider that more exact aDEV
can be measurable by adding the development time inconsideration of the error than our current system.
The experimental result indicated that the adequate
value of the threshold to calculate the aDEV should
be determined based on the development environmentof users. We are planning to propose automatic setting
method of the threshold value, based on the type of ﬁlesor a development environment.
Hackystat is a monitoring environment for collecting
development log of each user[10]. This environment cancollect a history of user control such as editing, opening,and closing of a ﬁle by adding a plug-in called ”sensor”
to a development environment, such as eclipse. Although
the hackystat can collect development log more detailed
than our system, there is a problem that the development
environment which can be used, is restricted.
We are going to apply these metrics to the educational
methods in SDPBL in order to improve PM technique.

[5] M. Unterkalmsteiner, T. Gorschek, A. Islam, C. Cheng,
R. Permadi, and R. Feldt, “Evaluation and measurementof software process improvement-a systematic literature
review,” IEEE Transactions on Software Engineering,,
vol. 38, no. 2, pp. 398 – 424, 2011.
[6] Dropbox Inc., “Dropbox,” https://www.dropbox.com/.
[7] “Dropbox4j,” https://github.com/Frostman/dropbox4j.
[8] “Dwr
easy
ajax
for
http://directwebremoting.org/dwr/index.html.

java,”[9] “jqplot,” http://www.jqplot.com/.
[10] P. Johnson, “Requirement and design trade-offs in hackystat: An in-process software engineering measurement
and analysis system,” in Proc. of 1st Int. Symposium on
Empirical Software Engineering and Measurement, IEEE
Computer Society Press, 2007.

VII. C ONCLUSION
In this research, we proposed two kinds of student
experience metrics, an accumulated LOC and an accumulated development time to visualize and compare theeffort of each student in SDPBL quantitatively. The online
storage service made it possible to collect the development
logs of each student for measuring such metrics at a low
cost. We are planning to improve the accuracy of theaccumulated development time with using an estimation
method of operation start time based on the accumulated
LOC.
ACKNOWLEDGMENT
This research was partially supported by the Japan Ministry of Education, Science, Sports, and Culture [Young
Scientists (B) (No.24700030)]
R EFERENCES[1] E. de Graaff and A. Kolmos, Management of Change Implementation of Problem-Based and Project-Based Learning
in Engineering. Sense Publishers, 2006.

469