Improvement of a Visualization Technique for the
Passage Rate of Unit Testing and Static Checking
and its Evaluation
Yuko Muto, Kozo Okano, and Shinji Kusumoto
Graduate School of Information Science and Technology, Osaka University
Suita, Japan{y-mutoh, okano, kusumoto}@ist.osaka-u.ac.jp
Abstract—Software visualization has attracted lots of attention.
The techniques fall into two categories: visualization of software
component relationships and visualization of software metrics.
We have already proposed a hybrid method based on both of
the two categories. The proposed method visualizes coincidence
between specification and implementation from two aspects:
static checking and ordinal testing by test suites. Each of theverification is performed in a method or function basis (unit
testing). In the method, each ratio of the coincidence is shown by
pie charts which represent classes of the target software. Whole
software is represented in a weighted digraph structure.
In this paper, we propose Priority Layout to emphasize
important classes , and implemented our method into a tool. We
have evaluated time in finding bug at source code and test cases
between using Priority Layout, ISOM Layout and uncomplicatedtables instead of graphs. As a result, time in finding bug at sourcecode and test cases by proposed graph are a half of it using table.
Keywords-unit testing; static checking; ESC/Java2; software
quality; visualization

I. I NTRODUCTION
Recently, visualization techniques for software play more
important roles according to increase of the size of software.
The techniques fall into two categories: visualization ofsoftware component relationship and visualization of software
metrics. The former approaches[1] often show program flows
as PDG (Program Dependency Graph). The latter approaches
include visualization of temporal sequence of software metrics
which helps analysis of software development aspects[2].
The granularity of the visualization target varies from code
segments to objects, class files, or libraries [3]. For objectoriented programs, class is one of suitable granularity. Paper[4] shows several relationships among classes.
ISO defines quality of software [5] including functionality
which means if given software satisfies required properties.
The functionality can be measured by ordinary unit testing,static checking and model checking. Ordinary unit testing
tests for a given module and its specification if the modulesatisfies the specification using sufficient amount of test suites.
Ordinary unit testing is usually performed as an early step ofsoftware tests. A major drawback of ordinary unit testing isthat the quality of results of the test sometimes depends on thequality of test suites used. If the coverage of the test suites islow, then some of properties cannot be tested.

On the other hand, static checking and model checking
approaches do not require executing the source code. These
approaches check statically via source code (or abstract modelof the source code, which models behavior of the source code).
One of the famous tools of static checking is ESC/Java2[6].
Its input is Java program annotated with JML (Java Modeling
Language) [7], [8], in DbC manner. It checks if the (behaviourof the) source code satisfies the property described in JML.
The quality of output also depends on the property itself as
well as that of standard libraries used for ESC/Java2. Other
drawback of ESC/Java2 is that it is not easy to understand its
outputs for especially novices because outputs are text based.
Relationships among classes should be visualized.
Therefore, a hybrid approach is considered. In paper[9], wehave proposed a hybrid method based on both of the twocategories. The method visualizes coincidence between specification and implementation from two aspects: ordinary testing(by test suites) and static checking. Each of the verificationis performed in a method or function basis (unit testing). In
the method, the ratios of the coincidence are shown by pie
charts which represent classes of the target software. Whole
software is represented in a weighted digraph structure. The
prototype tool runs as a plug-in of Eclipse. We have evaluatedthe availability of the proposed method by applying the tool to
two kinds of software: Warehouse Management Program anda telephone directory management program. As a result, weconclude that the proposed method shows informative results.
GraphTrace[4] has proposed a visualization method for
OOP, to understand dynamic behavior of the program. The
target language is OO LispD. It has structural and behavioral
views, which show tree views of class inheritance and method
call structures using the source code and runtime execution
information. Paper[4] provides some case-study examples
showing that visualization is useful.
Some papers [3] and [10] have proposed visualization methods for components of software. Paper [3] also summarizes
that visualization is performed in several views: static views
which show abstract structure of programs, and dynamic
views which depict dynamic traces of programs. Recently,
quality of software becomes important. Few papers, however,provide visualization of the quality of software. Our approachovercomes such a weakness.

Check’n’Crash[11] combines ordinary unit testing and static
checking. It automatically identifies faults as the following
flows. ESC/Java2 produces some counter examples. Then
test suites are automatically produced based on the counterexample, which are used in unit testing to identify faults. It
is effective in a sense that it produces only suitable test suites
for suspect faults.
The above work performs ordinary unit testing after static
checking. Paper [12] provides the opposite way. Tests cannot
find corner case bugs. The method in [12] firstly performs
testing to the target and obtains its coverages. Secondly
it performs static checking on the complement part of thecoverages. Thus, the static checking can be applied to a limitedarea of the target source code; it gains scalability. It, however,misses the bugs which are passed by the tests but could be
detected by static checking alone.
In Paper [13], authors provide and evaluate quality of contracts by means of mutation testing. From their observation, it
is important to support developers by tool such as providing
quality of assertion because there are some differences amongindividuals. They develop a tool which generate mutants fromtest target program for Java and C. They defined mutation coverage depending on lower and upper bound of completeness of
JML assertions to evaluate contracts quality. The upper boundis the number of detected mutants divided by the number of
non-equivalent mutants, and the lower bound is the numberof detected mutants divided by the number of all mutants.
They evaluate quality of contracts in the following process:generating random test cases, executing the test cases into
target program, generating meta-mutants by their tool andexecuting the test cases into them, computing upper and lower
bound from difference in results between the target programand mutants.
Comparing with using mutations, our approach has anadvantage in execution time. Our approach needs just an
execution of unit testing and static checking without advanced
preparing such as generating mutants.
In this paper, we propose Priority Layout to emphasize
important classes, and implemented our method into a tool.
We have evaluated time in finding bug at source code andtest cases between using Priority Layout, ISOM Layout anduncomplicated tables instead of graphs. As a result, time infinding bug at source code and test cases by proposed graphs
are a half of using table.
The paper is organized as follows. Section II provides thedefinitions of some words as preliminary. Section III will show
our proposed method. We give an overview of our prototype
tool in Section IV, followed by experiments and discussion in
Section V and VI. Finally Section VII concludes the paper.
II. P RELIMINARY
This section gives some definitions and explanation on unit
testing and static checking.
A. Unit Testing(Ordinary) unit testing is performed onto each module ofa given software. Conventionally the testing is performed by

Fig. 1.

Main class and Person class with JML

test suites.
Well-known metrics on unit testing includes statement
coverage, branch coverage, condition coverage, and so on.
These coverages are used for metrics for quality of test suites
themselves as well as that of results of unit testing.
JUnit, JCoverage, djUnit and others are tools for unit testing.
JUnit is the de facto standard framework for unit testing forJava. Its test case codes are described in TestCase class; it
executes the class in order to perform unit testing. The tool
indicates whether a test method passes or fails, i.e., whether thetest target method called by the test method makes expectedbehavior or not. JCoverage[14] enumerates coverages. It does
not depend on specific tools such as JUnit. It creates coveragereports on specified metrics and granularity. djUnit is anassistant tool to integrate JUnit and JCoverage as eclipse
plugin. It requires test cases like JUnit, and exports coveragereports according to JCoverage 1 .
B. Static Checking
JML (Java Modeling Language)[7], [8] is a specification
language used for annotation into Java programming. Basedon DbC (Design by Contract)[15], we can give assertions suchas invariants, pre-conditions and post-conditions for a method.
Figure 1 shows an example of JML annotation. Person class
has a f ullN ame field and a setter method, setF ullN ame.
Method setF ullN ame has a pre-condition that nm is neither
Null nor null String, thus, the annotation of line 12 is given.
Also the ensure clause at line 13 gives the post-condition
which means that f ullN ame has the same value to nm.
ESC/Java2[16] is a static checking tool which verifies
whether the source code satisfies the annotation described
in JML for each method. In theory neither soundness nor
completeness is guaranteed, however it efficiently finds bugs
in normal usage. It is one of the useful tools in the sense of
a light-weight formal approach. It supports Java version 1.4.
1 However, it uses modified version of JCoverage; the value of the report
sometimes differs from the original one.

Fig. 2.

Visualization of static checking for Main class and Person class

We give an example for the metrics using Figure 2.
From the output by ESC/Java2, we can infer that constructor and the method setF ullN ame are valid, however method getF amilyN ame is not valid. Therefore
Mpassed (P erson) = 2 and M (P erson) = 3, respectively.
Cs (P erson) is calculated as 0.66.
B. Definition of Caller-Callee Relation

Some of major libraries have been annotated in JML or builtin. It requires a Java source code and outputs a result as text
messages which describes passed or failed for each property
and each method. If it reports failed, its counter-example isalso generated.
III. O UR P ROPOSED M ETHOD
Figure 2 visualizes the result of static checking for Figure
1. Caller-callee relation of the given target program is shownin a digraph, where each node and each edge represents aclass and the caller-callee relation, respectively. Each nodealso represents a pie-chart which gives a passage rate of
the corresponding class. The passage rate is evaluated basedon unit testing and static checking. The weight of an edge
corresponds to the number of method calls relating to theclasses. We use caller-callee relation instead of class hierarchy
relation used typically in class diagram, because in this paper,
we focus on modular verification/testing, where properties ofclasses or methods and their relations are important. Of course,such a structure can be visualized using a similar way of ours.
Additionally, we defined Priority Layout to help find importantclasses.
A. Definition of Passage Metrics
Here, we have to think the following four kinds of metrics:(1) metrics for the quality of the test suites, (2) metrics of thequality of contracts, (3) metrics for the results of ordinary unit
testing, and (4) that for the results of static checking. In thepaper, we focus on the metrics for (1),(3) and (4). We discuss
the metrics for (2) later.
1) Passage Rate for Results of Unit Testing: We adopt
statement coverage as a coverage of unit testing. The reason
why is the following: statement coverage is simple and easy
to calculate; the value of branch coverage generated by djUnit
is different from the original value; and condition coverage is
not supported by JCoverage.
Our approach utilizes djUnit because it covers both of (1)and (3). The coverage report by djUnit contains only code
coverage data without acceptance of test methods, however,developers can recognize the acceptance of test method because JUnit view attached by djUnit provides that information.
2) Passage Rate for Results of Static Checking: Let
Mpassed (A) and M (A) be the number of passed methods ina class A, and the total number of methods in a class A,
respectively. The coverage of static checking for the class A
is defined as:
Cs (A) = Mpassed (A)/M (A).

(1)

If some method m1 appears statically in a method m2 as amethod call statement , we say m2 calls m1 . If a method inclass A calls some method in class B, we say class A calls
class B. Let nAB be the number of every call, such that class
A calls class B. We say class A calls class B nAB times.
In Figure 1, Main class calls constructor of Person class inline 3, setF ullN ame method in line 4 and getF amilyN ame
method in line 5. Thus, Main class calls Person class threetimes.
C. Priority Layout
We propose Priority Layout to emphasize important classes.
Figure 3 demonstrates Priority Layout. Some graph layouts
are proposed to avoid overlapping nodes and edges, suchas Circle Layout and ISOM Layout[17]. ISOM Layout iseffective for dense graphs because it is based on a competitive
learning strategy. Our approach requires a layout based on
characteristics of classes or caller-callee relationships because
the more important classes should be attracted more.
1) Importance: Let G = (V, E) be a graph, where V and E
denote sets of nodes and edges, respectively. Edge eij denotes
an edge from node vi to vj . Let w be weight of an edge.
w(eij ) = 1 if caller-callee relationship of from node vi to vj
exists. Otherwise, we let w(eij ) be 0. IN (vi ) is defined asa group of directed edges end with vi . Weight of node vi isdefined as the sum of a weight of a directed edge eki end with
vi . Importance is the weight of node. Also, importance of vi
equals the number of callees which are called by vi .
Definition 3.1 (Importance):∑w(vi ) =w(eki )
(2)eki ∈IN (vi )2) Layout: Classes with higher importance are allocated in
the more superior region of the graph. When classes have thesame importance, they are in a row at regular intervals.
IV. I MPLEMENTATION
Here, we give simple descriptions on our prototyped tool.
Please refer [9] for details. The tool is implemented as a plugin of Eclipse. The size of the program is about 2000 LOC
without comments, with 14 packages and 33 classes.
Figure 4 shows the screen shot of the tool. The tool has
Main View and Method View. Main View displays the digraph.
Here, a node corresponds to a class. The coverage is displayed
as a pie chart. If some class calls some other class many times,
the corresponding edge is drawn in thick style. Method View
displays detail method information on the selected class at
Main View.

TABLE I
G ROUP
Group
A
B
C
D

Fig. 3.

Example for Priority Layout

Fig. 4.

Screen shot

V. E XPERIMENTS
In order to evaluate our proposed method, we asked ten
examinees to use our tool.
A. GQM approach
We adopt GQM approach[18] to evaluate our tool. Figure 5
shows the result of applying GQM. From the result we choose
two metrics:• Response time to find bugs at source code
• Response time to find defects of test cases
In addition, questionnaire includes:
Q1
How much do you think comparing two graphs helps
you find bug?
Q2
Comparing between Priority Layout and others(ISOM Layout or table), which do you like?
Examinees must answer the questions in one to five point. For
instance, for Q1, answering score 1 means that our approachnever help finding a bug, answering score 5 means it can help
finding a bug. For Q2, answering score 1 means that Priority
Layout is worse than ISOM Layout or simple tables, answering
score 5 means that Priority Layout is better than others.
B. Methodology
We experimented in the following process. First, we explained how to use our tool to examinees and they practice
exercise by our tool. After that, they try to find bugs in source

Method used in Problem 1
ISOM Layout
Priority Layout
Table
Priority Layout

Method used in Problem 2
Priority Layout
ISOM Layout
Priority Layout
Table

code and its test cases for two problems and answer the time
to solve them and some questionnaires.
The below enumeration details the way to find bugs using
our tool.
1) Display unit testing result into view.
2) Display static checking result into view.
3) Find buggy method comparing unit testing and static
checking
4) Reason why the method is failed in static checking byreading specification
5) Find shortage line of test cases
There are ten examinees, who are members of our laboratory studying software engineering. They consist of a Ph.D
candidate, six master students and three undergraduates. They
understand Java grammar, unit testing and static checking. Allof them utilize Eclipse usually.
We gave some items to examinees: Java source code with
sufficient JML, test cases and specification written in Japanese.
We implemented Multiple Table View to compare our graphpropose and simple method. Multiple Table View contains
class names, passage rates of unit testing and static checking.
It does not have caller-callee relationship data.
Table I shows assigning examinees to groups. Group A andB compare between ISOM Layout and Priority Layout, andGroup C and D compare between Priority Layout and table
instead of graphs.
C. Problems
Target programs to solve are based on Warehouse Management Program[19]. We altered its source code and test cases
to be buggy. Warehouse Management Program consists ofseven classes: ContainerItem, Customer, Item, ReceptionDesk,
Request, Storage and StockState. Its test suites sizes 800 LOC.
addItem method of AppMain has a bug. addItem method
calls constructor of Item class without checking parameters.
On the other hand, in Figure 6, the constructor of Item class
has a pre-condition that parameter name must not be nulland amount must be greater than zero. Thus, addItem method
violates the pre-condition of constructor of Item class. addItem
method can behave unanticipatedly and cause troubles when
it is executed.
D. Result and Observation
Table II shows the result of time to detect buggy method,
excluding wrong answer and who cannot find the bug within
deadline. Time required to find the bug using Priority Layout
is the shortest, followed by using ISOM Layout and using
simple tables in this order. Using Priority Layout reduces theFig. 5.

Result of applying GQM Model

TABLE IIR ESPONSE T IME TO DETECT BUGGY METHOD
MethodISOM Layout
Priority Layout
Table

Fig. 6.

TABLE IIIR ESPONSE T IME TO DETECT INSUFFICIENT TEST CASE

Avg. of Time (min)
12.9
10.79
21.48

Item

MethodISOM Layout
Priority Layout
Table

Avg. of Time (min)
13.66
14.9
26.39

compare between graphs of unit testing and graphs of static
checking. Table II and the result of Q2 conclude that Priority
Layout improves visibility of the graphs for developers.
Since Priority Layout reduces response time in finding bugs
at source code, the examinees realize that Priority Layout isbetter than another layout.
The followings are some comments from the examinees:
The graph makes it easy to understand dependency ofclasses• It enables the users to easily realize classes to focus• It takes much time to understand the program structure• Visualizing dependency of methods is required• It must provide reason why methods were failed by static
checking.
•

time by 16 % of ISOM Layout, and using both of graphs result
half the time of using table. Thus, for this example program,usefulness of our proposed method was observed about finding
the bug in source code.
Table III summarizes the time required to find the bug intest cases. On the other hand, time to find the bug in test
cases, there is little difference between using Priority Layout
and ISOM Layout. The reason should be that our proposed
technique visualizes classes instead of methods. Let Tt be time
to find the bug in test cases, Tc be time to find buggy test class,and Tm be time in find buggy test method. These variables
have the following relation:
Tt = Tc + Tm .
Finding bugs in test cases occurs after finding bugs in sourcecode because we had an instruction to do so. A class andits test class have a one-to-one correspondence in the target
program, Hence,
Tc = 0andTt = Tmhold. Tm must be similar between using Priority Layout andISOM Layout because none of layout provide to help findmethod.
The average of answers are 3.5 and 4.3 for Q1 and Q2,
respectively. The result of Q1 shows that it is useful to

To quicken finding bugs, last two comments demonstrate theneed more information on methods.
E. Threats to Validity
1) Internal Validity: There is little difference in ability
among the groups. The examinees learned software development because they are members of software design laboratory.
They are divided into four groups evenly mixed their grades.
Among the examinees, there are few habituation for thetarget program and our tool because they read the target
program for the first time and they have never utilized thetool.
2) External Validity: The universality of the program is low
because we use small size program in the experiment. However, the program is a typical to manage inventories, it includes
the essence of similar kind of real systems. Consequently, ourtool is useful in these types of programs.
Answers of the questionnaire have low generality because
they are subjective data. Also, the reliability of the results forQ1 and Q2 is low. This is caused by the reason: although
the scores in the questionnaires are ordinary scale, the data isaveraged.

Hence, we need to perform more experiment. The example
includes comparing the response time for large size program
between using our tool and using eclipse without our tool.
VI. D ISCUSSION
We need metrics to specify the quality of given JML
statements. We have researched past papers, however, we findfew suitable existing coverage or metrics for JML. Thus, we
devise a new metric, called Variable Coverage. In general,
assertions are conditions on program variables. For example,
pre-condition and post-condition assert that parameters andreturn value (and/or some field variables) of the method meetthe conditions, respectively. In a similar way, Class Invariant
asserts invariant conditions for field variables while the object
is alive. Hence, we can regard the ratio of variables with its
condition as coverage. We consider Variable Coverage for amethod as a metric on its parameters, return value and related
field variables. Variable Coverage consists of Parameter Coverage, Return Value Coverage and Field Variables Coverage.
These coverages are used in combination. For example, fora typical post-condition, Return Value Coverage and Field
Variable Coverage are used.
Parameter Coverage is the ratio by the number of usedparameters in the pre-condition to that of all parameters.
Return Value Coverage means whether post-condition holds
return value or not. The result must be 0% or 100%.
Field Variable Coverage is the ratio by the number of usedfield variables in conditions to that of all field variables. Field
variables are classified into mutable and immutable in themethod. If a variable must change, post-condition would use
the variable. For the other variables, Pure or Invariant should
hold them.
VII. C ONCLUSION
In this paper, we improved and evaluated the visualization
method for software quality using unit testing and static
checking. We proposed Priority Layout, and performed evaluation using examinees. The results show that our approachcontributes to reduce the time in finding bugs at source codeand test cases.
Future work includes improving information on methods tofind easily actual bug location. Furthermore, we will researchand evaluate what we said in section VI, quality of the test
suites and annotations in JML.
ACKNOWLEDGMENT
This work is being conducted as a part of Stage Project,
the Development of Next Generation IT Infrastructure, supported by Ministry of Education, Culture, Sports, Science andTechnology, as well as Grant-in-Aid for Scientific ResearchC(21500036).
R EFERENCES[1] T. Miyake, Y. Higo, S. Kusumoto, and K. Inoue”, “”masu: A
metrics measurement framework for multiple programing languages[in japanese]”,” The IEICE transactions on information and systems(Japanese edition). D, vol. 92, no. 9, pp. 1518–1531, 2009. [Online].
Available: ”http://ci.nii.ac.jp/naid/110007361123/”[2] S. Morisaki and K. Matsumoto, “Toward optimized collection and visualization of software metrics for progress sharing in offshore softwaredevelopment project,” In Proc. of the 2nd Workshop on Accountability
and Traceability in Global Software Engineering (ATGSE2008), pp. 3–4,
2008.
[3] W. Lowe, M. Ericsson, J. Lundberg, and T. Panas, “Software
comprehension - integrating program analysis and software
visualization,” 2002. [Online]. Available: http://citeseerx.ist.psu.edu/
viewdoc/summary?doi=10.1.1.100.3818[4] M. F. Kleyn and P. C. Gingrich, “Graphtrace—understanding objectoriented systems using concurrently animated views,” in OOPSLA’88: Conference proceedings on Object-oriented programming systems,
languages and applications. New York, NY, USA: ACM, 1988, pp.
191–205.
[5] ISO, “Software engineering-product quality-part 1 : Quality model,”ISO/IEC : 9126-1:2001, 2001. [Online]. Available: http://ci.nii.ac.jp/
naid/10024638597/[6] P. Chalin, “Early detection of jml specification errors using esc/java2,”
in SAVCBS ’06: Proceedings of the 2006 conference on Specificationand verification of component-based systems. New York, NY, USA:ACM, 2006, pp. 25–32.
[7] C. Yoonsik and P. Ashaveena, “Specifying and checking method callsequences of java programs,” Software Quality Journal, vol. 15, no. 1,
pp. 7–25, March 2007. [Online]. Available: http://dx.doi.org/10.1007/
s11219-006-9001-4[8] L. Burdy, M. Huisman, and M. Pavlova, “Preliminary designof bml: a behavioral interface specification language for java
bytecode,” in Proceedings of the 10th international conference on
Fundamental approaches to software engineering, ser. FASE’07.
Berlin, Heidelberg: Springer-Verlag, 2007, pp. 215–229. [Online].
Available: http://portal.acm.org/citation.cfm?id=1759394.1759418[9] Y. Muto, K. Okano, and S. Kusumoto, “A visualization technique forsoftware quality using unit testing and static checking with caller-callee
relationship,” in Proceedings of International Conference on Advanced
Software Engineering, 2011.
[10] A. Gonzalez, R. Theron, A. Telea, and F. J. Garcia, “Combined visualization of structural and metric information for software evolution
analysis,” in IWPSE-Evol ’09: Proceedings of the joint international andannual ERCIM workshops on Principles of software evolution (IWPSE)
and software evolution (Evol) workshops. New York, NY, USA: ACM,
2009, pp. 25–30.
[11] C. Csallner and Y. Smaragdakis, “Check ’n’ crash: combining static
checking and testing,” in ICSE ’05: Proceedings of the 27th international
conference on Software engineering. New York, NY, USA: ACM, 2005,
pp. 422–431.
[12] V. Vipindeep and P. Jalote, “Efficient static analysis with path pruning
using coverage data,” in WODA ’05: Proceedings of the third international workshop on Dynamic analysis. New York, NY, USA: ACM,
2005, pp. 1–6.
[13] T. Knauth, C. Fetzer, and P. Felber, “Assertion-driven development:
Assessing the quality of contracts using meta-mutations,” in Proceedingsof the IEEE International Conference on Software Testing, Verification,
and Validation Workshops. Washington, DC, USA: IEEE Computer
Society, 2009, pp. 182–191. [Online]. Available: http://portal.acm.org/
citation.cfm?id=1547559.1548252[14] jcoverage ltd., “Jcoverage,” http://www.jcoverage.com/.
[15] B. Meyer, Object-oriented software construction (2nd ed.).
Upper
Saddle River, NJ, USA: Prentice-Hall, Inc., 1997.
[16] C. Flabagan, K. Rustan, M. Leino, M. Lillibridge, G. Nelson, J. B.
Saxe, and R. Stata, “Extended static checking for java,” Proc. of the
ACM SIGPLAN 2002, pp. 234–245, 2002.
[17] K. T, “The self-organizing map,” Proceedings of the IEEE, pp. 1464–
1480, 2002.
[18] V. Basili, “Using measurement to build core competencies in software,”Seminar sponsered by Data and Analysis Center for Software, 2005.
[19] M. Owashi, K. Okano, and S. Kusumoto, “Design of warehousemanagement program in jml and verification with esc/java2 [injapanese],” The IEICE transactions on information and systems(Japanese edition) D, vol. 91, no. 11, pp. 2719–2720, 2008. [Online].
Available: http://ci.nii.ac.jp/naid/110007380947/