Identifying Key Attributes of Projects that Affect the Field Quality ofCommunication Software
Nahomi Kikuchiyz, Osamu Mizunozand Tohru Kikunoz

kikuchi386@oki.co.jp, fo-mizuno, kikunog@ics.es.osaka-u.ac.jp

yOki Electric Industry Co., Ltd., Japan
zGraduate School of Engineering Science, Osaka University, Japan

Abstract
In this paper we identify key attributes of projects in which
the number of problem reports after release is remarkablein the communication software. After several interviews tosoftware project managers, we derived candidate attributesof importance to the projects. To find out the most influential
attributes of the projects, we conducted statistical analysis
using a set of metrics data related to the candidate attributes
and the number of problem reports after release. As a result, we successfully found that two metrics concerning theorigin’s quality and the changes in specification are usefulto estimate the field quality.

(2) Each project used similar programming languages (forinstance C, C++, etc.).
In order to find characteristics of projects, we classified
these projects into two groups: Good and Fair. To findcharacteristics of the Fair projects, we hold several interviews
with a number of project managers from both groups. Then,we examined and presented a list of candidate attributes ofimportance to the projects’ post-release quality. To identify
influential attributes of the projects, we conducted statisticalanalysis using a set of metrics data related to the candidate
attributes and the numbers of problem reports.

2
1 Objective and Approach
The quality of communication software is usually measured by the problem reports after release in the field. Thatis, the quality is only assessed by the resultant product. Actually there exist many studies on the quality of software
products[1, 2, 3]. Generally, review and test activities are effective to acquire better quality. Some of them discussed themethodologies of review and test[1, 2], and others proposed
statistical models[3] to analyze how the quality of software
increases. They, however, focused on the development ofgeneral software(rather than specific domain software), andthey did not consider explicitly the deadlines of projects.
The aim of this paper is to investigate the common attributes of projects whose field quality was not good. In thispaper, we don’t consider projects that did not finish withinpredetermined dates. The reason is that for such projects,
no reliable data are collected during development. It is thus
impossible to analyze those projects statistically.
Satisfaction of customers is then evaluated by field quality
after release. In addition, quality after the release is reflectedon problem reports after the release in the field. Thus, we
judge success of a project using the number of problem
reports originated by customers after release for one year
(called problem reports for short).
We chose twenty-four projects in a company that havecommonly the following characteristics:(1) Each project was completed within the schedule.

Communication Software Development

In the development of communication software, there exist several types of software development such as reusing oradapting from existing software, a previous version of thecurrent product, commercial off-the-shelf software (COTS),
and commercial or reuse libraries. The pre-existing software,
which is used as development basis, is generally called theorigin[6]. We then call a software product that was developed using pre-existing origins Prior work. Other type of asoftware product is newly developed software without using
pre-existing one, and is called New work.
Prior work is further divided into Modified, Adopted andReuse according to development activities. “Modified” partof software is developed based on pre-existing software withmany changes on design and codes. On the other hand,"Adapted" and “Reuse” part of software are developed withalmost no design and code change. In “Adapted” part ofsoftware, the design is reviewed and program is tested to alimited extent. In "Reuse" part of software, the design isneither reviewed nor tested at all.
The development process is a sequential software engineering life-cycle model according to the waterfall model.
Software requirement is determined in the analysis phase anddevelopment plan is constructed in the planning phase. Then
implementation is performed in the design phases(basic,functional and detail), the programming phase and the test
phases(unit, module and system). In each phase, software
work products and documents are created and reviewed toremove defects. For the purpose of project management,development reports are written at the end of each phase.

A software project manager describes development reports
and evaluates the progress and status to see if the project may
proceed to the next phase.
While other related systems such as hardware andfirmware are developed at the same time, the software partis sometimes asked for the change of interface specification in certain timing(for example, during the design phase).
When a specification is changed, some modules may also bechanged. Such changes of specification are likely to happen
frequently according to the stability of related systems.
Since the software development process is usually performed concurrently with the related systems’ development,
it is also difficult to verify performance specifications, suchas peak performance at real operational situation, in designphases. Thus, insufficient specifications suddenly appear in
the test phase, and they often require changes in the design
specifications.

(a) A problem concerning quality of the origin (Prior work)a-1) A fairly number of problems are found in the origin itself after release.
a-2) The development activity which aims to improve
quality of the origin is not enough.
a-3) The development activity to check the quality ofthe origin is not enough.
(b) A problem concerning test and review activityb-1) Review and test are done with little consideration
to the technical matter specific to communicationsoftware.
b-2) Activity of source code review and a unit test is
scarce in some projects.
(c) A problem concerning project management and planning

3 Interviewc-1) There is much delay before development plan andreport are constructed. Moreover, reporting to thequality assurance department is given a rather low
priority.

3.1 Categorization of projects
In order to distinguish projects whose quality is not considered as satisfactory, we adopted the metrics F t1 . (Weintroduce other related metrics for statistical analysis in Section 4.)
F t1 : the number of problem reports for twelve months
after releaseWe call projects with unsatisfactory value for F t1 Fair
project. Furthermore, we classified Fair projects into two
subgroups Poor and Average based on the degree of theirpost-release quality. Classification rules are summarized asfollows:c-2) The development plan is not detailed enough tobe applied actually.
c-3) The development activity is emphasized andproject management tends to become scanty.
(d) A problem concerning specification changed-1) Changes of design specification occur even in programming phase and test phases. Software part issometimes asked for change of interface specification by other related systems.

( Good project ; Ft < nn Average project ; n  Ft < nFair project

1Poor1project ; n2  Ft111d-2) Oversights appear in the operational condition
and/or environment, which should be included in
the design specification.

2

These numbers n1 and n2 were determined by having
discussion with the quality assurance group.
For the twenty-four target project, we applied the rule toclassify them. Fourteen projects are then in Good group andten in Fair. In more precise, four projects are in Average andsix projects are in Poor.

3.2 Qualitative results of interview
After the classification of projects in subsection 3.1, wechecked common attributes of both Good group and Fairgroup using the documents of the development plan andreport written by an individual project. However, we failed
to find common attributes in each group Good and Fair. Inorder to know the projects’ detailed activities and trends thatare not described in the reports, we thus hold interviews
with software projects managers of both projects of Fair andGood.
The following (a)–(d) were confirmed as the candidates ofcommon attributes that make a project Fair.

4

Statistical Analysis using Logistic Model

4.1

Metrics for Process Quality

We established the detailed evaluation items by which
individual project activities are evaluated. The items are selected to express the qualitative attributes discussed in subsection 3.2.

total (the number of problem reports of the product)

FQF t1: the number of problem reports for twelve months
after releaseorg (the number of problem reports of the origin’s product)

FQ

F o1

: the number of origin’s problem reports for twelve
months after release per origin’s software size

Qorg (evaluation of activity for improving the origin’s quality): quality of the origin part of software
O2 : level of examining the origin’s quality in designphase
O3 : level of improving activity for the origin’s quality
in design and test phases
O1

Q

rev (evaluation of review activity): level of consideration to exceptional cases in design review
R2 : timeliness of design review performed
R1Qtest (evaluation of test activity): level of consideration to exceptional cases andoperational conditions in test activity
T2 : contents of source code review and unit test
T1Qmng (evaluation of management)
M1
M2

Q

: adequateness of creating and revising plan
: adequateness of information included in plan

spec (evaluation of specification change): the timing that the specification is fixed
S2 : preciseness of the specification fixed
S3 : understandability of requirements and feasibilityof design specification
S1

4.2 Logistic Regression Analysis
We evaluated twenty-four projects by the above Q-metrics
data. For each project data, we summed up each sub-itemin the metric group and got a value of each metric. We thenused logistic regression model to analyze the relationships
among the metrics and field quality.
The response variable we use to validate the metrics isbinary, i.e. was a project’s field quality Good or Fair. The
value of this response variable is evaluated by the logistic
regression model to be described. We took six metrics,
F Qorg , Qspec , Qrev , Qtest , Qorg and Qmng as candidatesof explanatory variables. For each explanatory variable, thevalue was obtained by summing up the sub-items’ score ineach item. Table 1 represents a part of measured metrics data.
In Table 1, classification shows the actual result evaluatedby the rule in subsection 3.1 using the value of F Qtotal . As
mentioned before, out of the twenty-four projects, fourteen
projects are Good and ten projects are Fair.
A multivariate logistic regression model is given in thefollowing formula:b +b x + +bn xn111

j 1 1 1 n ) = 1 + b +b x + +bn xn

E (Y x1 ;;xe 0e 0

1 111

111

In this formula, response variable Y represents whether
a project is Good or Fair, and explanatory variables xi are

Table 1. Metrics data

Qspec Qrev
44442
66
6
6
4

4
4
222
8
7
46
2

Qtest Qorg
5
55
6
5
15
16
14
14
9

2
2
2
2
2
7
7
7
96Qmng

111

111

111

111

111

111

111

24

115

24

1125

Good

Project
1234
5
6
7
8
910

17
171616
16
14
7
15
16
3ClassGoodGoodGoodGood
Good
Fair
Fair
Fair
Fair
Good

supposed to be selected appropriately. Thus, the value ofconditional probability E (Y jx1 ; 1 1 1 ; xn ) is found.
Based on the 24 projects’ data, we estimated coefficientsbi by means of step-wise selection and created a logistic
reduction model. As a result, two variables Qspec and Qorg
were selected as significant.
Deviance was shown to be 19.978 and the degree of freedom was 13. Therefore, the goodness of fit for the model was
shown to be good. Moreover, we checked the significance ofthe model by likelihood ratio test. As a result, it was shown
to be somewhat significant p 0 value = 0:0095(< 0:01).
Thus, the model was shown to be significant with significance level 1%.
Using the model for classification and the 24 project data,when E > 0:5, the project is classified as Fair and, otherwise,
as Good. Twenty projects out of 24 were predicted correctlyby the model. If we take into account individual project
predicted incorrectly, three projects out of four were Average
projects. Although the proposed model did not completely
predicted, predictions using our Qorg and Qspec metricsappear to be useful for the field quality of communication
software.

References[1] L. J. Arthur, Improving Software Quality - An Insider’s Guide toTQM -, John Wiley & Sons, 1993.
[2] R. G. Ebenau and S. H. Strauss, Software Inspection Process,
McGraw-Hill, 1993.
[3] J. D. Musa, A. Iannino and K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGraw-Hill, 1987.
[4] O. Mizuno, T. Kikuno, Y. Takagi and K. Sakamoto, “Characterization of Risky Projects based on Project Managers’ Evaluation,”
Accepted to ICSE2000.
[5] V. R. Basili, L. C. Briand and W. L. Melo, “A Validation of ObjectOriented Metrics as Quality Indicators,” IEEE Trans. on SoftwareEng., vol. 22, no.10, pp.751-761, 1996.
[6] R. E. Park, “Software Size Measurement: A Framework for Counting Source Statements”, CMU/SEI-92-TR-20, 1992.