# Towards Use-Cases Benchmark

Bartosz Alchimowicz, Jakub Jurkiewicz, Jerzy Nawrocki, Miroslaw Ochodek

► To cite this version:

Bartosz Alchimowicz, Jakub Jurkiewicz, Jerzy Nawrocki, Miroslaw Ochodek. Towards Use-Cases Benchmark. 3rd Central and East European Conference on Software Engineering Techniques (CEESET), Oct 2008, Brno, Czech Republic. pp.20-33, 10.1007/978-3-642-22386-0_2 . hal-01572532

HAL Id: hal-01572532

https://inria.hal.science/hal-01572532

Submitted on 7 Aug 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Towards Use-Cases Benchmark*

Bartosz Alchimowicz, Jakub Jurkiewicz, Jerzy Nawrocki, Mirosław Ochodek

Poznan University of Technology, Institute of Computing Science,
ul. Piotrowo 3A, 60-965 Poznań, Poland
{Bartosz.Alchimowicz, Jakub.Jurkiewicz, Jerzy.Nawrocki,
Miroslaw.Ochodek}@cs.put.poznan.pl

**Abstract.** In the paper an approach to developing a use-cases benchmark is presented. The benchmark itself is a referential use-case-based requirements specification, which has a typical profile observed in real projects. To obtain this profile an extensive analysis of 432 use cases coming from 11 projects was performed. Because the developed specification represents those found in real projects, it might be used in order to present, test, and verify methods and tools for use-case analysis. This is especially important because industrial specifications are in most cases confident, and they might not be used by researchers who would like to replicate studies performed by their colleagues.

**Keywords:** Use cases, Requirements engineering, Metrics, Benchmark

## 1 Introduction

Functional requirements are very important in software development. They impact not only the product but also test cases, cost estimates, delivery date, and user manual. One of the forms of functional requirements are use cases introduced by Ivar Jacobson about 20 years ago. They are getting more and more popular in software industry and they are also a subject of intensive research (see e.g. [2,3,8,9,20]). Ideas presented by researches must be empirically verified (in best case, using industrial data) and it should be possible to replicate a given experiment by any other researcher [19]. In case of use cases research it means that one should be able to publish not only his/her results but also the functional requirements (use cases) that has been used during the experiment. Unfortunately, that is very seldom done because it is very difficult. If a given requirements specification has a real commercial value, it will be hard to convince its owner (company) to publish it. Thus, to make experiments concerning use cases replicable, one needs a benchmark that would represent use cases used in real software projects.

In the paper an approach to construct use-cases benchmark is presented. The benchmark itself is the use-cases-based requirements specification, which has a

* This research has been financially supported by the Polish Ministry of Science and Higher Education grant N516 001 31/0269.
typical profile observed in requirements coming from the real projects. To derive
such a profile an extensive analysis of 432 use cases was performed.

The paper is organised as follows. In Section 2 a model of use-cases-based
requirements specification is presented. This model is further used in Section 3,
which presents an analysis of the use-cases, coming from eleven projects. Based
on the analysis a profile of the typical use-case-based specification is derived,
which is used to create a benchmark specification in Section 4. Finally, a case
study is described in Section 5, which presents a typical usage of the benchmark
specification in order to compare tools for use-case analysis.

## 2 Benchmark-oriented model of use-case-based specification

Although use cases have been successfully used in many industrial projects (Neil
et al. [16] reported that 50% of projects have their functional requirements pre-
sented in that form), they have never been a subject of any recognisable stan-
dardisation. Moreover, since their introduction by Ivar Jacobson [13] many ap-
proaches for their development were proposed. Russell R. Hurlbut [11], gathered
fifty-seven contributions concerning use cases modeling. Although, all approaches
share the same idea of presenting actor's interaction with the system in order to
obtain his goal, they vary in a level of formalism and presentation form. Thus
it is very important to state what is understood by the term use-cases-based
requirements specification.

To mitigate this problem a semi-formal model of use-cases-based require-
ments specification has been proposed (see figure 1). It incorporates most of the
best-practices presented in [1,5].

It still allows to create specification that contains only scenario (or non-
structured story) and actors, which is enough to compose small but valuable
use-case, as well as extend it with other elements, for example extensions (using
steps or stories), pre- and post-conditions, notes or triggers.

Unfortunately, understanding the structure of specification is still not enough
in order to construct a typical use-cases-based specification. What is missing, is
the quantifiable data concerning the number of model-elements and proportions
between them<fnref n="1" />. In other words, one meta-question has to be asked for each
element of the model - "how many?".

## 3 Analysis of use-cases-based specifications structure

In order to discover the profile of typical-specification, data from various projects
was collected and analysed. As a result a database called UCDB (*Use Cases
Database*) [6] has been created. At this stage it stores data from 11 projects
with the total number of 432 use cases (basic characteristics of requirements
specifications as well as projects descriptions are grouped in table 1).

<footnote n="1">¹ A number/proportion of any element of the model will be further addressed as a
**property** of the specification</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Use-Cases-based functional requirements specification model</figcaption></figure>

All of the specifications have been analysed in order to populate the model with the information concerning average number of its elements occurrence and some additional quantifiable data.

One of the interesting findings is that 79.9% of the main-scenarios in the use cases consist of 3-9 steps, which means that they fulfil the guidelines proposed by Cockburn [5]. What is more 72.9% of the analysed use cases are augmented with the alternative scenarios (extensions). There are projects which contains extensions for all steps in main scenario, however on the average use case contains 1.5 extension. Detailed information regarding number and distributions of steps in both - main scenario and extensions, is presented in figure 2.

Another interesting observation, concerning the structure of use cases, is that the sequences of steps performed by the same actor, frequently contains more than one step. What is even more interesting this tendency is more visible in case of main actor's steps (37.3% of main actor's steps sequences are longer than one step, and only 19.4% in case of secondary actor - in most cases system being built). This is probably because actions performed by main actor are more important, from the business point of view. This is contradict to the concept of transactions in use cases presented by Ivar Jacobson [12]. Jacobson enumerated four types of actions which may form together use-case transaction. Only one
<table><caption>**Table 1.** Analysed projects requirements-specifications (*origin: industry - project developed by software development company, s2b - project developed by students for external organisation, external - specification obtained from the external source which is freely accessible through the Internet - this refers to two specifications: UKCDR [21], PIMS [18]; projects D and K come from the same organisation*)</caption><thead><tr><th rowspan="2">ID</th><th rowspan="2">Specification Language</th><th rowspan="2">Origin</th><th colspan="4">Number of use cases</th><th rowspan="2">Description</th></tr><tr><th>All</th><th>Business</th><th>User</th><th>Sub-function</th></tr></thead><tbody><tr><td>Project A</td><td>English</td><td>S2B</td><td>17</td><td>0%</td><td>76%</td><td>24%</td><td>Web &amp; standalone application for managing members of organization</td></tr><tr><td>Project B</td><td>English</td><td>S2B</td><td>37</td><td>19%</td><td>46%</td><td>35%</td><td>Web-based Customer Relationship Management (CRM) system</td></tr><tr><td>Project C</td><td>English</td><td>External</td><td>39</td><td>18%</td><td>44%</td><td>33%</td><td>UK Collaboration for a Digital Repository (UKCDR)</td></tr><tr><td>Project D</td><td>Polish</td><td>Industry</td><td>77</td><td>0%</td><td>96%</td><td>4%</td><td>Web-based e-government Content Management System (CMS)</td></tr><tr><td>Project E</td><td>Polish</td><td>S2B</td><td>41</td><td>0%</td><td>100%</td><td>0%</td><td>Web-based Document Management System (DMS)</td></tr><tr><td>Project F</td><td>Polish</td><td>Industry</td><td>10</td><td>0%</td><td>100%</td><td>0%</td><td>Web-based invoices repository for remote accounting</td></tr><tr><td>Project G</td><td>English</td><td>External</td><td>90</td><td>0%</td><td>81%</td><td>19%</td><td>Protein Information Management System (PIMS)</td></tr><tr><td>Project H</td><td>Polish</td><td>Industry</td><td>16</td><td>19%</td><td>56%</td><td>25%</td><td>Integration of two sub-systems in ERP scale system</td></tr><tr><td>Project I</td><td>Polish</td><td>Industry</td><td>21</td><td>38%</td><td>57%</td><td>5%</td><td>Banking system</td></tr><tr><td>Project J</td><td>Polish</td><td>Industry</td><td>9</td><td>0%</td><td>67%</td><td>33%</td><td>Single functional module for the web-based e-commerce solution</td></tr><tr><td>Project K</td><td>Polish</td><td>Industry</td><td>75</td><td>0%</td><td>97%</td><td>3%</td><td>Web-based workflow system with Content Management System (CMS)</td></tr></tbody></table>

of them belongs to a main actor (user request action). The rest of them are system actions: validation, internal state change, and response. It might look that those actions should frequently form together longer sequences. However, 80.6% of steps sequences performed by system consisted of single step only. The distributions and number of steps in main actor's sequences are presented in figure 3.

If we look deeper into the textual representation of the use cases, some interesting observation concerning their semantic might be made. One of them regards the way use-cases authors describe validations actions. Two different approaches are observed. The most common is to use extensions to represent alternative system behaviour in case of verification-process failure (46.6% of extensions have this kind of nature). Second one, and less frequent, is to incorporate validation actions into steps (e.g. *System verifies data*). This kind of actions are observed only in 3.0% of steps. There is also yet another interesting semantic structure
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Scenarios lengths in analysed use cases a) histogram presents the number of steps in main scenario (data aggregated from all of the projects), b) box plot presents the number of steps in main scenario (in each project), c) histogram presents the number of steps in extension (data aggregated from all of the projects), d) box plot presents the number of steps in extension (in each project, note that project E was excluded because it has all alternative scenarios written as stories)</figcaption></figure>

used to represent alternative execution paths - conditional clauses (which in fact, are rather deprecated). Fortunately this kind of statements are observed only in 3.2% of steps (and they occur intensively only in 2 projects).

Analysed properties can be classified into two separate classes. The first one contains properties which are observed in nearly all of the projects, with comparable intensity. Those kind of properties are seem to be independent from the specification they belong to (from its author's writing style). The second class is an opposite one, and includes properties which are either characteristic only for a certain set of use cases or their occurrence in different specifications are in between of two extremes - full presence or marginal/none. Such proper-
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Distributions of actors steps-sequences lengths in **analysed use cases**,
a) histogram presents the length of the steps sequences performed by main actor,
b) histogram presents length of the steps sequences performed by secondary actor
- e.g. System</figcaption></figure>

ties are project-dependent. More detailed description of analysed requirements specifications is presented in table 2.

# 4 Building referential specification

Combining use-cases model and average values coming from the analysis, a profile of the typical use-cases-based requirements specification can be derived. In such specification all properties would appear with the typical intensity. In other words, analysis performed on such document could be perceived as an every-day task for use-cases analysis tools.

There might be at least two approaches to acquire instance of such specification. The first one would be to search for the industrial set of use cases, which would fulfil given criteria. Unfortunately, most of industrial documents are confidential, therefore they could not be used at large scale. The second, obvious approach would be to develop such specification from scratch. If it is built according to the obtained typical-specification profile, it might be used instead of industrial specification. Since it would describe some abstract system it can be freely distributed and used by anyone.

Therefore, we would like to propose approach to develop an instance of the referential specification for the benchmarking purpose. The document is available at the web site [6] and might be freely used for further research.
<table><caption>**Table 2.** Use-Cases Database analysis according to the presented requirements specification model (see section 2)</caption><thead><tr><th rowspan="3">Property</th><th rowspan="3">Overall</th><th colspan="12">Project</th></tr><tr><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>I</th><th>J</th><th>K</th></tr><tr><th>17</th><th>37</th><th>39</th><th>77</th><th>41</th><th>10</th><th>90</th><th>16</th><th>21</th><th>9</th><th>75</th></tr></thead><tbody><tr><td>Number of use cases</td><td>432</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Number of steps in main scenario</td><td>4.87</td><td>4.76</td><td>4.92</td><td>2.95</td><td>4.34</td><td>5.78</td><td>3.90</td><td>5.10</td><td>3.38</td><td>4.33</td><td>3.67</td><td>6.36</td></tr><tr><td>Use cases with extensions</td><td>2.48</td><td>0.97</td><td>1.46</td><td>2.69</td><td>2.25</td><td>1.26</td><td>1.20</td><td>2.41</td><td>0.50</td><td>2.01</td><td>1.32</td><td>3.25</td></tr><tr><td>Number of extensions in use case</td><td>72.9%</td><td>94.1%</td><td>51.4%</td><td>41.0%</td><td>55.8%</td><td>92.7%</td><td>100%</td><td>68.9%</td><td>81.3%</td><td>90.5%</td><td>55.6%</td><td>98.7%</td></tr><tr><td>Number of steps in extension</td><td>1.50</td><td>1.29</td><td>0.57</td><td>0.95</td><td>0.92</td><td>1.63</td><td>1.00</td><td>1.28</td><td>2.94</td><td>2.33</td><td>1.00</td><td>2.69</td></tr><tr><td>Steps with validation actions</td><td>SD</td><td>1.84</td><td>0.77</td><td>0.60</td><td>1.72</td><td>1.12</td><td>0.62</td><td>0.00</td><td>1.30</td><td>2.98</td><td>1.58</td><td>1.58</td><td>2.91</td></tr><tr><td>Extensions which are validations</td><td>Mean</td><td>2.51</td><td>1.80</td><td>2.52</td><td>1.00</td><td>1.45</td><td>N/A</td><td>1.10</td><td>2.77</td><td>2.30</td><td>1.64</td><td>1.44</td><td>3.09</td></tr><tr><td>Main actor's steps sequence length in main scenario</td><td>Mean</td><td>1.62</td><td>0.84</td><td>0.87</td><td>0.00</td><td>0.59</td><td>N/A</td><td>0.32</td><td>1.87</td><td>0.65</td><td>0.67</td><td>0.53</td><td>1.80</td></tr><tr><td>Secondary actor's steps sequence length in main scenario</td><td>Mean</td><td>3.0%</td><td>3.7%</td><td>5.5%</td><td>11.3%</td><td>1.8%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>27.8%</td><td>17.6%</td><td>3.0%</td><td>0.0%</td></tr><tr><td>Use cases with additional description</td><td>Mean</td><td>46.6%</td><td>9.1%</td><td>76.2%</td><td>64.9%</td><td>63.4%</td><td>40.3%</td><td>100%</td><td>50.4%</td><td>78.7%</td><td>89.8%</td><td>100%</td><td>15.3%</td></tr><tr><td>Number of use cases with sub-scenario</td><td>1</td><td>62.7%</td><td>42.3%</td><td>93.1%</td><td>88.6%</td><td>37.0%</td><td>61.9%</td><td>87.5%</td><td>91.7%</td><td>47.4%</td><td>52.4%</td><td>50.0%</td><td>52.6%</td></tr><tr><td>Use cases with pre-conditions</td><td>2</td><td>20.1%</td><td>23.1%</td><td>3.5%</td><td>6.8%</td><td>34.8%</td><td>36.1%</td><td>12.5%</td><td>0.0%</td><td>0.0%</td><td>19.1%</td><td>42.9%</td><td>16.4%</td></tr><tr><td>Use cases with post-conditions</td><td>3</td><td>9.8%</td><td>26.9%</td><td>3.5%</td><td>4.6%</td><td>16.3%</td><td>0.0%</td><td>0.0%</td><td>8.3%</td><td>31.6%</td><td>23.8%</td><td>7.1%</td><td>12.9%</td></tr><tr><td>Use cases with triggers</td><td>4</td><td>3.9%</td><td>7.7%</td><td>0.0%</td><td>0.0%</td><td>6.5%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>21.1%</td><td>0.0%</td><td>0.0%</td><td>8.6%</td></tr><tr><td>Steps with conditional clauses</td><td>>4</td><td>3.5%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>5.4%</td><td>2.1%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>4.8%</td><td>0.0%</td><td>9.5%</td></tr><tr><td>Number of steps with reference to use cases</td><td>1</td><td>80.6%</td><td>82.6%</td><td>96.3%</td><td>72.2%</td><td>67.9%</td><td>100%</td><td>100%</td><td>90.0%</td><td>50.0%</td><td>12.5%</td><td>62.5%</td><td>72.1%</td></tr><tr><td>Number of extensions with scenario</td><td>2</td><td>13.3%</td><td>17.4%</td><td>3.7%</td><td>22.2%</td><td>29.6%</td><td>0.0%</td><td>0.0%</td><td>8.0%</td><td>16.7%</td><td>43.8%</td><td>37.5%</td><td>14.8%</td></tr><tr><td>Number of extensions with stories</td><td>3</td><td>3.3%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>2.0%</td><td>33.3%</td><td>25.0%</td><td>0.0%</td><td>7.4%</td></tr><tr><td>Number of extensions with stories</td><td>4</td><td>1.9%</td><td>0.0%</td><td>0.0%</td><td>5.6%</td><td>2.5%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>4.1%</td></tr><tr><td>Number of extensions with stories</td><td>>4</td><td>0.9%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>18.8%</td><td>0.0%</td><td>1.6%</td></tr><tr><td>Number of use cases with sub-scenario</td><td>38.0%</td><td>0.0%</td><td>0.0%</td><td>100%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>100%</td><td>56.3%</td><td>100%</td><td>0.0%</td><td>6.7%</td></tr><tr><td>Number of steps in sub-scenario</td><td>13.7%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>32.5%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>44.2%</td></tr><tr><td>Use cases with pre-conditions</td><td>1.98</td><td>0.00</td><td>0.00</td><td>0.00</td><td>3.20</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>1.09</td></tr><tr><td>Use cases with post-conditions</td><td>1.69</td><td>N/A</td><td>N/A</td><td>N/A</td><td>2.02</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>0.29</td></tr><tr><td>Use cases with triggers</td><td>48.5%</td><td>82.4%</td><td>100%</td><td>0.0%</td><td>0.0%</td><td>97.6%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>23.8%</td><td>77.8%</td><td>0.0%</td></tr><tr><td>Steps with conditional clauses</td><td>10.6%</td><td>0.0%</td><td>0.0%</td><td>97.4%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>68.8%</td><td>57.1%</td><td>0.0%</td></tr><tr><td>Number of steps with reference to use cases</td><td>36.3%</td><td>100%</td><td>100%</td><td>100%</td><td>0.0%</td><td>100%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>Number of steps with reference to use cases</td><td>3.2%</td><td>1.2%</td><td>1.1%</td><td>17.4%</td><td>0.3%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>9.0%</td><td>9.0%</td></tr><tr><td>Number of extensions with scenario</td><td>6.3%</td><td>0.0%</td><td>0.5%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>25.6%</td><td>14.2%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>6.1%</td><td>33.3%</td></tr><tr><td>Number of extensions with stories</td><td>71.9%</td><td>27.3%</td><td>100%</td><td>8.1%</td><td>87.3%</td><td>0.0%</td><td>100%</td><td>100%</td><td>100%</td><td>100%</td><td>100%</td><td>100%</td><td>100%</td></tr><tr><td>Number of extensions with stories</td><td>28.1%</td><td>72.7%</td><td>0.0%</td><td>91.9%</td><td>12.7%</td><td>100%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td></tr></tbody><tfoot><tr><td colspan="13">Requirements specification independent</td></tr><tr><td colspan="13">Requirements specification dependent</td></tr></tfoot></table>
## 4.1 Specification domain and structure

It seems that large number of tools for the use-cases analysis have their roots in the research community. Therefore the domain of the developed specification should be easy to understand especially for the academia staff. One of the well-recognised processes at universities is a **students admission process**. Although developed specification will describe hypothetical process and tools, it should be easy to understand for anyone who have ever participated in a similar process.

Another important decision concerns the structure of the specification (number of actors, use cases and business objects). Unfortunately the decision is rather arbitral, because number of use cases, actors and business objects may depend on the size of the system and level of details incorporated into its description. In this case a median number of use cases and actors from the UCDB set has been used as a number of use cases and actors in the constructed specification. The final specification consist of **7 actors** (Administrator, Candidate, Bank, Selection committee, Students Management System, System, User), **37 use cases** (3 business, 33 user, and 1 sub-function level), and **10 business objects**.

## 4.2 Use cases structure

A breadth-first rule has been followed in order to construct referential specification. Firstly, all use cases were titled and augmented with descriptions. Secondly, all of them were filled with main scenarios and corresponding extensions. During that process all changes made in specification were recorded in order to check conformance with the typical profile. This iterative approach was used until the final version of the specification has been constructed. Its profile is presented in table 3, in comparison to the corresponding average values from the UCDB projects analysis (main actor's step sequences lengths for referential specification, are additionally presented in figure 4). The most important properties are those which are observed in all of the specifications (specification independent). In case of those metrics most values in the referential document are very close to those derived from the analysis. This situation differs in case of properties which were characteristic only for certain projects (or variability between projects were very high). If the constructed specification is suppose to be a typical one, it should not be biased by features observed only in some of the industrial specifications. Therefore dependent properties were also incorporated into the referential use cases, however they were not a subject of the tuning process.

# 5 Benchmarking use cases - case study

Having the example of the typical requirements specification, one can wonder how it could be used. Firstly, researchers who construct methods and tools in order to analyse use cases often face the problem of evaluating their ideas. Using some specification coming from the industrial project is a typical approach, however, this cannot lead to the conclusion that the given solution would give
<table><caption>**Table 3.** Referential specification profile in comparison to the average profile derived from the UCDB use-cases analysis</caption><thead><tr><th colspan="2">Property</th><th>Referential<br>Specification<br>Admission System</th><th>Observed in Use-<br>Cases Database</th></tr></thead><tbody><tr><td rowspan="2">Number of steps in main<br>scenario</td><td>Mean</td><td>4.89</td><td>4.87</td></tr><tr><td>SD</td><td>1.47</td><td>2.48</td></tr><tr><td colspan="2">Use Cases with extensions</td><td>70.3%</td><td>72.92%</td></tr><tr><td rowspan="2">Number of extensions in use<br>case</td><td>Mean</td><td>1.50</td><td>1.50</td></tr><tr><td>SD</td><td>0.69</td><td>1.84</td></tr><tr><td rowspan="2">Number of steps in extension</td><td>Mean</td><td>2.49</td><td>2.51</td></tr><tr><td>SD</td><td>2.12</td><td>1.62</td></tr><tr><td colspan="2">Steps of validation nature</td><td>2.2%</td><td>3.0%</td></tr><tr><td colspan="2">Extensions of validation nature</td><td>46.2%</td><td>46.6%</td></tr><tr><td colspan="2">Steps with conditional clauses</td><td>0.6%</td><td>3.2%</td></tr><tr><td rowspan="5">Main actor's steps sequence<br>length in main scenario</td><td>1</td><td>62.3%</td><td>62.7%</td></tr><tr><td>2</td><td>21.3%</td><td>20.1%</td></tr><tr><td>3</td><td>9.8%</td><td>9.8%</td></tr><tr><td>4</td><td>3.3%</td><td>3.9%</td></tr><tr><td>&gt;4</td><td>3.3%</td><td>3.5%</td></tr><tr><td rowspan="5">Secondary actor's steps sequence<br>length in main scenario</td><td>1</td><td>80.7%</td><td>80.6%</td></tr><tr><td>2</td><td>12.9%</td><td>13.3%</td></tr><tr><td>3</td><td>4.8%</td><td>3.3%</td></tr><tr><td>4</td><td>1.6%</td><td>1.9%</td></tr><tr><td>&gt;4</td><td>0.00%</td><td>0.9%</td></tr></tbody></table>

the same results for other specifications. The situation looks different when the typical specification is considered, then researchers can assume that their tool would work for most of the industrial specifications. Secondly, analysts who want to use some tools to analyse their requirements can have a problem to choose the best tool that would meet their needs. With the typical specification in mind it can be easier to evaluate the available solutions and choose the most suitable one. To conclude the example of the typical requirements specification can be used:

- to compare two specifications and assess how the given specification is similar to the typical one

- to assess the time required for a tool to analyse requirement specification

- to assess the quality of a tool/method

- to compare tools and choose the best one

In order to demonstrate the usage of the constructed specification we have conducted a case study. As a tool for managing requirements in form of use cases we chose the UC Workbench tool [15], which has been developed at Poznan University of Technology since the year 2005. Additionally we used the set
<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Distributions of actors steps-sequences lengths in **referential specification**, a) histogram presents the length of the steps sequences performed by main actor, b) histogram presents length of the steps sequences performed by secondary actor - e.g. System</figcaption></figure>

of tools for defects detection [4] to analyse the requirements. The aim of the mentioned tools is to find potential defects and present them to analyst during the development of the requirements specification. The tools are based on the Natural Language Processing (NLP) methods and use Standford parser [7] and/or OpenNLP [17] tools to perform the NLP analysis of the requirements.

## 5.1 Quality analysis

In order to evaluate the quality of the used tools, a confusion matrix will be used [10] (see table 4).

<table><caption>**Table 4.** Confusion matrix</caption><thead><tr><th rowspan="2" colspan="2"></th><th colspan="3">Expert answer</th></tr><tr><th>Yes</th><th>No</th><th></th></tr></thead><tbody><tr><th rowspan="2">System answer</th><th>Yes</th><td>TP</td><td>FP</td></tr><tr><th>No</th><td>FN</td><td>TN</td></tr></tbody></table>

The symbols used in table 4 are described below:
- T (true) - tool answer that is consistent with the expert decision

- F (false) - tool answer that is inconsistent with the expert decision

- P (positive) - system positive answer (defect occurs)

- N (negative) - system negative answer (defect does not occur)

On the basis of the confusion matrix, following metrics [14] can be calculated:

- Accuracy (AC) - proportion of the total number of predictions that were correct. It is determined using the equation 1.

$$AC = \frac{TP + TN}{TP + FN + FP + TN} \qquad (1)$$

- True positive rate (TP rate) - proportion of positive cases that were correctly identified, as calculated using the equation 2.

$$TP \ rate = \frac{TP}{TP + FN} \qquad (2)$$

- True negative rate (TN rate) is defined as the proportion of negatives cases that were classified correctly, as calculated using the equation: 3.

$$TN \ rate = \frac{TN}{TN + FP} \qquad (3)$$

- Precision (PR) - proportion of the predicted positive cases that were correct, as calculated using the equation 4.

$$PR = \frac{TP}{TP + FP} \qquad (4)$$

The referential use-case specification was analysed by the mentioned tools (to perform the NLP processing Stanford library was used) in order to find 10 types of defects described in [4]. Aggregated values of the above accuracy-metrics are as follows:

- AC = 0.99

- TP rate = 0.96

- TN rate = 0.99

- PR = 0.82

This shows that the developed tools for defects detection are rather good, however, the precision could be better and the researchers could conclude that more investigation is needed in this area.

If the researchers worked on their tool using only defect-prone or defect-free specifications the results could be distorted and could lead to some misleading conclusions (e.g. that the tool is significantly better or that it gives very poor outcome). Having access to the typical specification allows the researchers to explore main difficulties the tool can encounter during working with the industrial specifications.
## 5.2 Time analysis

One of the main characteristics of a tool being developed is its efficiency. Not only developers are interested in this metric, but also for the users - e.g. when analysts have to choose between two tools which give the same results, they would choose the one which is more efficient. However, it can be hard to assess the efficiency just by running the tool with any data, as this may result with distorted outcome. Use-cases benchmark gives the opportunity to measure the time, required by a tool to complete its computations, in an objective way. It can be also valuable for researchers constructing tools to consider using different third-party components in their applications.

For instance, in case of the mentioned defect-detection tool there is a possibility of using one of the available English grammar parsers. In this case study two of them will be considered - Standford parser and OpenNLP. If one exchanges those components the tool will still be able to detect defects, but its efficiency and accuracy may change. Therefore, the time analysis was performed to assess how using those libraries influence the efficiency of the tool. The overall time required to analyse the typical specification<fnref n="2" /> for the tool using Stanford parser was 54.37 seconds and for the one with OpenNLP it was 31.21 seconds. Although the first time value seems to be large, the examined tool needed on average 290 ms to analyse a single use-case step, so it should not be a problem to use it in the industrial environment. Of course, when comparing efficiency, one have to remember that other factors (like the quality of the results, memory requirements or the time needed for the initialization of the tools) should be taken into account, as they can also have significant impact on the tool itself. Therefore, time, memory usage and quality analysis is presented in table 5. This summarised statistics allow researchers to choose the best approach for their work.

Although this specification describes some abstract system, it shows the typical phenomenon of the industrial requirements specifications. The conducted case study shows that the developed referential use-case specification can be used in different ways by both researchers and analysts.

## 6 Conclusions

In the paper an approach to create a referential use-cases-based requirements specification was presented.

In order to derive such a specification 432 use cases were analysed. It has proved that some of the use-case properties are project-independent (are observed in most of the projects). They have been used for creating the referential specification. On the other hand, there is also a number of properties which depend very much on the author.

<footnote n="2">² Tests were performed for the tools in two versions: with Stanford and OpenNLP parsers. Each version of the tools analysed the referential specification five times, on the computer with Pentium Core Duo 2.0GHz processor and 2GB RAM.</footnote>
<table><caption>**Table 5.** Case-study results (*tools are written in Java, so memory was automatically managed – garbage collection*)</caption><thead><tr><th rowspan="2"></th><th colspan="3">Summarised for all components</th><th colspan="3">English grammar parser only</th></tr><tr><th>Overall processing time [s]</th><th>Mean time needed to analyse one step [s]</th><th>Maximal memory utilization [MB]</th><th>Overall processing time [s]</th><th>Startup time [s]</th><th>Initial memory usage [MB]</th><th>Memory usage while processing single element [KB]</th></tr></thead><tbody><tr><td>Stanford</td><td>54.37</td><td>0.29</td><td>212</td><td>37.44</td><td>0.8</td><td>27</td><td>547</td></tr><tr><td>OpenNLP</td><td>31.21</td><td>0.17</td><td>339</td><td>14.58</td><td>15.6</td><td>226</td><td>3925</td></tr><tr><th colspan="8" style="text-align: center;">Quality</th></tr><tr><td></td><th>AC</th><th>TP rate</th><th>TN rate</th><th>PR</th><td></td><td></td><td></td></tr><tr><td>Stanford</td><td>0.99</td><td>0.96</td><td>0.99</td><td>0.82</td><td></td><td></td><td></td></tr><tr><td>OpenNLP</td><td>0.99</td><td>0.84</td><td>0.99</td><td>0.79</td><td></td><td></td><td></td></tr></tbody></table>

In order to present potential usage of the benchmark specification, a case study was conducted. Two sets of tools for defect detection in requirements were compared from the point of view of efficiency and accuracy.

In the future it would be beneficial to extend the use-case database with more use-cases coming from commercial projects. This would require an iterative approach to updating the typical-profile as well as the referential specification.

**Acknowledgments.** We would like to thank Piotr Godek, Kamil Kwarciak and Maciej Mazur for supporting us with industrial use cases.

This research has been financially supported by the Polish Ministry of Science and Higher Education under grant N516 001 31/0269.

## References

1. Steve Adolph, Paul Bramble, Alistair Cockburn, and Andy Pols. *Patterns for Effective Use Cases*. Addison-Wesley, 2002.

2. B. Anda and D.I.K. Sjøberg. Towards an inspection technique for use case models. *Proceedings of the 14th international conference on Software engineering and knowledge engineering*, pages 127–134, 2002.

3. B. Bernardez, A. Duran, and M. Genero. Empirical Evaluation and Review of a Metrics-Based Approach for Use Case Verification. *Journal of Research and Practice in Information Technology*, 36(4):247–258, 2004.

4. Alicja Ciemniewska, Jakub Jurkiewicz, Jerzy Nawrocki, and Łukasz Olek. Supporting use-case reviews. In *10th International Conference on Business Information Systems*, LNCS. Springer Verlag, April 2007.

5. A. Cockburn. *Writing effective use cases*. Addison-Wesley Boston, 2001.

6. Use Case Database. http://www.ucdb.cs.put.poznan.pl.
7. Marie-Catherine de Marneffe, B. MacCartney, and Ch. D. Manning. Generating typed dependency parses from phrase structure parses. In *Proceedings of the EACL workshop on Linguistically Interpreted Corpora (LINC)*, 2006.

8. C. Denger, B. Paech, and B. Freimut. Achieving high quality of use-case-based requirements. *Informatik-Forschung und Entwicklung*, 20(1):11–23, 2005.

9. S. Diev. Use cases modeling and software estimation: applying use case points. *ACM SIGSOFT Software Engineering Notes*, 31(6):1–4, 2006.

10. T. Fawcett. Roc graphs: Notes and practical considerations for data mining researchers. 2003.

11. R. Hurlbut. A Survey of Approaches for Describing and Formalizing Use Cases. *Expertech, Ltd*, 1997.

12. I. Jacobson. Object-oriented development in an industrial environment. *ACM SIGPLAN Notices*, 22(12):183–191, 1987.

13. Ivar Jacobson, Magnus Christerson, Patrick Jonsson, and Gunnar Overgaard. Object-oriented software engineering: A use case driven approach, 1992.

14. K. Krawiec and J. Stefanowski. *Uczenie maszynowe i sieci neuronowe*. Wydawnictwo Politechniki Poznańskiej, 2004.

15. Jerzy Nawrocki and Łukasz Olek. Uc workbench - a tool for writing use cases. In *6th International Conference on Extreme Programming and Agile Processes*, volume 3556 of LNCS, pages 230–234. Springer Verlag, Jun 2005.

16. CJ Neill and PA Laplante. Requirements Engineering: The State of the Practice. *Software, IEEE*, 20(6):40–45, 2003.

17. OpenNLP. http://opennlp.sourceforge.net.

18. PIMS. http://www.mole.ac.uk/lims/project/srs.html.

19. Forrest J. Shull, Jeffrey C. Carver, Siram Vegas, and Natalia Juristo. The role of replications in empirical software engineering. *Empirical Software Engineering*, 13(2):211–218, April 2008.

20. S.S. Somé. Supporting use case based requirements engineering. *Information and Software Technology*, 48(1):43–58, 2006.

21. UKCDR. http://www.ukoln.ac.uk/repositories/digirep/index/all_the_scenarios_and_use cases submitted #ukcdr.