Evaluating Design Rationale in Architecture
Marlies van Steenbergen

Floris Bex

Pim de Jong

Jan Martijn E. M. van der Werf

Sogeti Netherlands B.V.
Nieuwegein, The Netherlands
pim.de.jong@sogeti.com

Utrecht University
Utrecht, the Netherlands
j.m.e.m.vanderwerf@uu.nl

Utrecht University
Sogeti Netherlands B.V.
Nieuwegein, The Netherlands Utrecht, the Netherlands
f.j.bex@uu.nl
marlies.van.steenbergen@sogeti.com

Abstract—Although architecture is often seen as the culmination of design decisions, design rationale is a supposititious childin architecture documentation. Many architecture frameworks
and standards, like TOGAF and ISO/IEC 42010, recognize theimportance, but do not offer any support in the rationale process.
Recent initiatives have shown that simple means help in providing
more rationale. However, there are very few studies that give
evidence whether more rationale indeed leads to better quality.
In this paper, we propose a non-invasive method, the Rationale
Capture Cycle, that supports architects in capturing rationale
during the design process. Through a controlled experiment with
10 experienced architects, we observe the effectiveness of themethod in terms of design quality through different measures.
The results of our experiments show that: (1) better rationale isstrongly correlated with high quality, and (2) the test group with
our proposed method outperforms the control group.
Index Terms—design reasoning, design rationale, enterprisearchitecture, architecture documentation

as it explains the reasoning behind why the architecture isas it is [5]. Without proper explicit rationale, the architecture
degradates [6], and architectures become increasingly brittle
over time if documentation of decisions is not kept. For
example, architects may leave the organization, switch roles orswitch projects. Additionaly, architectural documentation withrationale makes it easier for architects to understand a design,especially if the architects were not involved in the original
design process: Tang et al. [5] found that architects strongly
agree on the premise that they cannot understand a design
effectively without its rationale.
In architecture design1 , many methods and techniques forreasoning about architecture designs have been proposed.
Examples of specific techniques are solution option generation,
constraint analysis, risk analysis, assumption analysis, andtrade-off analysis. The research and presentation of these techniques, however, is fragmented, and a single method that encompasses all of them is lacking. Frameworks like IBIS (IssueBased Information Systems) [7] and QOC (Questions-OptionsCriteria) [8] combine different concepts such as options, issues
and constraints and provide a language and accompanying
modelling tool to map design rationale. A major problem ofthis focus on “rationale mapping” is that it interferes with thedesign reasoning process: there is a cognitive overload that
results from having to learn and use such tools at the sametime as having to discuss and think about complex architecturaldesigns [5]. Although widely available, these frameworks are
therefore still underutilized in industry [5], [9], [10].
Architecture development methods such as the Togaf Architecture Development Method [11] also recognise the importance of careful design reasoning and include separate steps
for, for example, requirements analysis and cost-benefit analysis. Furthermore, architectural standards such as SysADL [12],GRASP [6] and ISO/IEC 42010 [1] mention the importance of
the architecture rationale, that is, the reasons underlying theimportant architecture design decisions. However, the focusof these methods and standards lies firmly on the descriptionof the enterprise or system itself, and not on the reasoning

Matthieu Brinkhuis
Utrecht University
Utrecht, the Netherlands
m.j.s.brinkhuis@uu.nl

I. I NTRODUCTION
Architectures, or more specifically architectural descriptions, refer to the fundamental elements of a system and how
they relate to comprise a perspective of the system in its context [1]. In the case of Enterprise Architecture the architecturaldescription hence concerns the organization or enterprise, its
IT systems, (functional) units, business processes and external
contacts such as customers and suppliers.
Architecture is often regarded as the culmination of design
decisions [2], [3]. Such design decisions are made during thedesign reasoning process [2], in which we have to identify
the context and requirements of the enterprise, formulate andstructure the design problem, think about possible solution options and make trade-offs between different options. Designingcomplex architectures for larger enterprises can be considered
a wicked problem [4]: the design problem is ill-defined; there
is no stopping rule to tell when an acceptable solution has beenreached; it is difficult to tell what is true or false in a design (atleast not before implementation); resolving one design issue
can give rise to other related design issues; and understanding
the problem depends on how the designer wants to solve it.
In the case of wicked design problems it is important tocapture the design rationales, the reasons underlying design
decisions. Properly documenting architectural design decisions
and their rationale is essential to any architectural description,

1 Note that here we look at both Enterprise Architecture and Software
Architecture, as many of the general frameworks, standards and techniques
for each of the architectural fields are very similar.

process or the product of this process, the design rationales,
which are only mentioned superficially. In other words, it is
not exactly elaborate how designers should reason and capture
rationale.
Recently, initiatives have been started to develop and evaluate normative methods that encourage and support the architecture design reasoning process and associated rationale
capture. For example, Razavian et al. [13] employ reflective
questions by an external observer, Schriek et al. [14] present
a card game, and Keil et al. [15] use checklists for decision
making. All studies show that with simple means, the rationale
process can be improved.
On the other hand, only very few studies study the benefit of
improving the rationale process. For example, Tang et. al. [16]show that architects produce, on average, better architecturesby explicitly reasoning about their design decisions. Theirstudy shows that design reasoning principles are applied more
effectively when provided with a systematic approach.
Under practicioners, there is the general perception that,
although design rationale should be captured in documentation, a proper methodology is lacking [5]. This observationis the starting point of our paper: “What is an effective
method to support the provision of design rationale in thearchitecture design process?”. To answer this question, westudy the existing body of knowledge in enterprise architectureand design reasoning, to develop a new method: the Rationale
Capture Cycle. To study the effect of the method, we perform
a controlled experiment with 10 experienced enterprise architects. We define effectiveness as the perceived quality of thesolutions delivered by the architect, and measure this in three
ways: (1) through peer review by the participating architects,(2) by analyzing the documentation on rationale usage, and (3)by a ranking of an external architect. Our results are twofold:(1) better rationale is strongly correlated with high quality, and(2) the test group outperforms the control group.
The remainder of this paper is structured as follows. First,
we discuss in Section II the setup of our research. Section III
provides a background in design rationale, and introduces ourtreatment method, the RCC. The results of the experiment arepresented in Section IV and discussed in Section V. Section VI
concludes the paper.
II. R ESEARCH D ESIGNThe ultimate objective of this study is to provide a means bywhich architecture designers can systematically employ design
reasoning. To this end, we apply design science [17]. First weconstruct a method through method engineering. Through acontrolled experiment we evaluate the effect of the method:
the method is given as treatment to the test group, whereas thecontol group works without the method. For the experiment,
we followed the guidelines by Wohlin et al. [18].
A. Method Design
To construct a method that supports the reasoning process,we followed the method association approach [19]. The final
method has been developed through two iterative cycles. In

the first cycle, we studied existing rationale methods andtechniques and architectural frameworks and languages. This
resulted in a set of requirements, both on the steps in themethod, and which rationale techniques could be employed ineach step. Next, the method has been validated and discussed
through semi-structured interviews with 10 architects. The
feedback we received from these interviews were then used
to improve and finetune the resulting method.
B. Experiment Design
To evaluate the effect of the method, we conduct a controlled experiment with two groups: a test group that receives
the method as treatment, and a control group who works
without [18]. We applied stratified sampling for assigning theparticipants to the groups, controlling the factors experience(m=30yr, SD=8), age (m=56yr, SD=7), and familiarity.
The experiments consists of two stages. In the first stage,
each participant has to individually solve a case where they
are asked to design in two hours a complete architecture, andprovide textual rationale for their reasoning process. First, thecase is introduced by one of the authors to all the participantsin a group session of about 30 minutes. Next, the participantsin the control group leave the session and start on theirsolution. In the mean time, the participants in the test groupare asked to use the treatment before they start to work on theirassignment. At the end of the design session, each participant
fills in a questionnaire about the reasoning process, the caseand the method. This stage results in 10 solutions to the case:five for each group.
In the next stage, each participant ranks three solutions bydividing 100 points to the different solutions, without creatinga tie. The design of the ranking satisfies the following criteria:
Each participant reviews at least a solution from bothgroups, not being their own;
• Each combination should be unique, i.e., no given ranking
occurs more than once;• Each solution is ranked an equal number of times, equallydistributed over the two groups;• In the ranking no cluster of solutions is disconnected fromthe others, i.e., a total order can be induced from thepartial orders formed by the individual rankings;•

To observe the effect of the treatment, we use four measures:
1) The total points given by the ranking of the architects,
to provide the relative design quality of the solutions;2) The documentation delivered by the participants iscounted, coded and analysed;
3) A full ranking of all solutions created by an external
enterprise architect;4) The results from the questionaire;
The coding schema used for the second measure is based on
the available literature on rationale techniques gathered during
the method design. Based on these measures, we evaluate
whether the proposed method is effective for capturing rationale, and for delivering better quality.

Table I
A RCHITECTURE RATIONALE TECHNIQUES , THEIR DEFINITION AS USED DURING THE CODING STAGE , AND ITS SOURCE
Rationale type
Design decision
Problem analysis
Constraint analysis
Assumption analysis
Option analysis
Benefit analysis
Weakness analysis
Trade-off analysis
Risk analysis
Evaluation
Reflection

Definition
Final decision made during the reasoning process.
Key issues and requirements in a design.
Limitations on the design, how they relate and materialize.
Unknown factors affecting the design, and their consequences.
Alternatives addressing the same design problem.
Benefits a design option can deliver to satisfy the requirements.
Weaknesses a design option have and their effect to the design.
Trade-offs to compare alternatives and its supporting rationales.
Unknown factors with negative implications, on a design option andhow they impact the design.
Feedback loop to rethink critical elements in the rationale.
Verification of the architect’s own reasoning process.

C. Case Design
Participants are presented a fictional case of a Dutch bank.
The case describes a bank that started as an exclusive bankserving generations of prosperous families, but now wants to
expand its clientage. In this process, the bank is facing asurge of new clients that bring about new wishes and needs interms of the service. As the services provided by the bank areoutdated, the bank hires architects to provide goal architecture
models to sketch the ideal system landscape for them. The
architects need to provide an architecture description in which
their rationale must be documented.
The case has been designed such that it is simple and clear
enough to understand, and feasible to complete within 2 hours,
yet be intricate enough, so rationale can be present. A pilot
test with 4 master students from Utrecht University is held tovalidate the case and eliminate any ambiguous elements. The
pilot showed that the case is challenging, especially given thelimited amount of time. As architects have more experience,and constantly need to make decisions on what to pursue, we
decided to keep the length and level of difficulty.
III. C APTURING D ESIGN R ATIONALE
Design reasoning refers to the process of reasoning on
design issues and coming to a design decision [5]. It relieson critical and reflective thinking, evaluation, discussion andlogical inferencing. Key principles include problem structuring, assumption recognition, and trade-off analysis. The
product of design reasoning is design rationale, which canbe defined as the justification for a design decision. This
includes the entirety of explicit reasoning that was required tomake a decision, i.e. the explicit culmination of argumentativeknowledge to weigh and elect options.
A. Architecture Rationale
The Oxford Dictionary defines rationale as a “reason or
intention for a specific action”. In the context of architecture
design, rationale is considered to be “the reasons behinda design decision, the justifications for it, the alternatives
considered, trade-offs evaluated and argumentation that led tothe decision” [23]. An overview of the most frequently usedrationale techniques is presented in Table I.

Source
[20]
[5]
[20]
[5]
[21]
[20]
[22]

Design rationale can occur in various forms and types, depending on their use and current need [24]:• Argumentation based: the design rationale presents arguments that define the design. These arguments present
pros and cons for each alternative, option and issue.
• History based: the design rationale represents the design
history of a system. This includes the dimension of time,i.e., design decisions are included chronologically.
• Device based: the design rationale is based on a model ofthe device or system itself. The model can simulate thebehaviour of the system, from which the design rationale
can be deduced.
• Process based: the design rationale is completely interwoven with the design process itself and therefore guides
the format of the rationale.
• Active document based: the design rationale is predefinedand already saved in the system. When an architect
designs an architecture, the design rationale system automatically generates rationale for the elements designedby the architect.
These types are by no means exhaustive and may simultaneously be true. For example, architecture documentation cancontain a full history of all options with their pros and cons,
and which options have been chosen. Which type of rationale
is used, also depends on how rationale is captured. Lee [23]identified five major methods to do so:• Reconstruction constitutes the production of rationale
by reasoning from existing knowledge (introspection).
Reconstruction is always post-mortem, i.e., completely
separated from the design process. As such is it nonintrusive to the design process itself.
• Record and Replay captures rationale during the design
process. Design problems are identified, alternatives are
considered and criteria and claims are defined whilst
design is taking place. This can occur through digitalmeans (forums, videoconferencing) or through regularface to face meetings.
• Methodological By-product: in this approach the rationale is produced as a logical by-product of following amethod in the architecture design process. This method
constitutes capturing the rationale. The idea is that the1. Define the problem.

Problem
7. Trade-off analysis.

Identify and describe the key
issue(s) in your design or therequirement you want to satisfy.

If a design cannot satisfy multiple
requirements and constraints
simultaneously, describe why you
made the design decision as
opposed to another.

8. End loop.

6. Risk analysis.

Evaluate the decision and reflect
on the reasoning process. Document
the final design decision.

Describe any potential factors that
can have negative implications on
the design.

Options
2. Explore options.
Explore and describe thevarious options and alternative
solutions on how the problem
or requirement can beaddressed.

5. Constraint analysis.
Describe any constraints that may
impede on the successful
implementation of your design.

3. Benefits & weaknesses.
Describe the benefits andweaknesses of your options andelaborate on how they may affectthe design.

4. Assumption analysis.
Recognize and describe any madeassumptions when expanding on thepotential options.

Figure 1. The Rationale Capture Cyclearchitect follows a method that supports generating rationale during the design process.
• Apprentice: the apprentice approach consists of the interaction between a designer and a computer system. The
system verifies design decisions made by the designer
and asks questions whenever an action is made which itdoes not understand.
• Automatic Generation: constitutes a system that produces
rationale from an existing rationale base. The systemanalyses a complete history of designs and defines the
how’s and why’s of the performed actions.
The main design philosophy revolves around creating amethod that captures rationale whilst least interfering with regular design activities. Therefore, the most successful rationale
capture approach is likely to be a methodological by-productapproach [23], combined with process- and argumentationbased rationale. Such combination should result in a feasible
and approachable method [24].
B. The Rationale Capture Cycle
Previously proposed methods for rationale capture were
ineffective due to a variety of reasons [5]. Methods are either(1) too limited in depth: offering no real handson support,(2) too complex: overly daunting for an average user, or(3) too specific: often inapplicable in various domains or
projects. These factors all contribute to the lack of industry
standards [5]. The Rationale Capture Cycle (RCC) is createdaround countering these three factors.

The sequence of the RCC is based on the Planning and
Problem-Solution Co-Evolution theory [20]. As early decisions heavily influence the process in which design activities
are carried out [20], designers should consider a high-level design plan first, i.e., one should identify the main requirements
and design issues first.
The rationale process is triggered through the identificationof a problem, such as the satisfaction of a requirement. For
this solution, many different options can be generated andexplored, which all need to be analysed. For each option, thearchitect should analyze the possible benefits and weaknesses,
which assumptions are foundational to the option, whichconstraints are either being put by the option, or hamper the
option, the risks involved and a final weighted decision basedon a proper trade-off analysis. For all activities, the architectshould evaluate and reflect to what extent the process has been
followed, and critically assess what could be improved to therationale, and the process itself. The RCC captures each ofthese activities in a cyclic manner.

IV. R ESULTS
In the controlled experiment, we observed three measures
to define a ranking: the points awarded by the participants,
the coding of the documentation, and the full ranking by theexternal architect (Table II).

Table IIR ANKINGS BY THE DIFFERENT MEASURES : POSITION , POINTS ,DOCUMENTATION AND EXTERNAL RANKING . E LEMENTS MARKED WITH ∗OR ∗∗ ARE RANDOMLY ASSIGNED , AS THEY HAVE THE SAME AMOUNT OF
POINTS .
Pos.
1.
2.
3.
4.
5.

Points
T4
T2
T3
C5
T1 ∗

Doc.
T4
T2
C5
T1
T3

Ext.
T4
T2
T3
C5
T1

Pos.
6.
7.
8.
9.
10.

Points
C1 ∗
T5 ∗∗
C2∗∗
C3
C4

Doc.
C2
T5
C1
C4
C3

Ext.
C1
T5
C2
C4
C3

Table IIIS OLUTIONS RANKED BASED ON THE ASSIGNED POINTS . E LEMENTS
MARKED WITH ∗ OR ∗∗ ARE RANDOMLY ASSIGNED , AS THEY HAVE THE
SAME AMOUNT OF POINTS .
Rank
1
2
3
4
5

Solution
T4
T2
T4
C5
T1∗

Points
160
130
115
110
95

Rank
6
7
8
9
10

Solution
C1∗
T5∗∗
C2∗∗
C4
C3

Points
95
90
90
60
55

A. Design Quality
The first measure to observe the design quality is the relative
ranking provided by the participants themselves. Recall thateach architect ranked three solutions by dividing 100 points tothe solutions, and that ties were not allowed. Each participant
received at least one solution from both groups. By summing
the points per solution, we obtain the relative ranking by theparticipants.
None of the participants gave a solution 0 points. Half ofthem used a 50-30-20 point scale. The other distributions were45-35-20 (two times), 75-20-5 and 65-20-15 (both one time).
The solutions of T1 and C1, and of T5 and C2 received anequal amount of points. To maintain a full ranking, these wererandomly assigned to their respective positions. This resultsin the ranking of Table III. A first observation is that threesolutions out of the top 4 are from the test group.
B. Rationale Documentation
The participants had to explicate their entire reasoning process in the documentation, including the justifications for theirdesign decisions. As a next step, we coded the architecture
documentation of each participant to distinguish the individual
rationale types in each document. The coding was performedby one of the authors, after which the results were validated
and discussed with two of the other authors to reach consensus.
The coding schema used is given in Table I. During the coding,we made an explicit distinction between identifying a rationale
technique, and describing it. For example, one can identify arisk, but not describe why it is a risk, or how the architecture
is affected. Additionally, we counted the number of words inthe documentation.
The coding shows that the participants of the test groupused 122 rationale elements, whereas the control group used
69 elements. These were individual elements of rationale,elaborating on why they made certain design decisions. In-

terestingly, there are three elements that have no occurrencein the documents across the various participants: evaluation,reflection, and assumption definition. Important to realize
is that this does not imply that these techniques have notbeen used, solely that in the documentation these were notdemonstrated explicitly.
C. Feedback by the Architects
At the end of the session, the architects of both groups wereasked to fill in a questionaire to give feedback and additional
thoughts on the RCC, the case, and the overall experiment.
Most questions regarding the RCC concerned added value,
effectiveness, readability, usability, intuitiveness, and ease ofuse.
a) Added value & effectiveness: There were mixed feelings about the value and usefulness of the architecture: twoarchitects found the case too small for using something likethe RCC, whereas three other architects claimed that the RCC
helped them to capture rationale more easily, especially since
the RCC made them capture assumptions that they otherwise
would not have made explicit. The RCC gave them structure
to the process which is “desperately needed”.
b) Readability & usability: Most architects found the
RCC easy to read and understand. One architect mentioned
the cycle was confusing, as there is no real end to the process.
c) Intuitiveness & ease: Most architects mentioned that
the RCC did not intrude on the design process, and that itwas easy to use. One architect found that the RCC asks for a
too high investment of time, which is only realistic if the casewould offer choice moments of larger size.
d) : The participants were also asked to share theirthoughts regarding the case. Out of all 10 architects, 4 mention
that the case did not accurately mimic a real life scenario,as the architect normally has a chance to ask and research
the scenario, which is inherent to the type of case provided.
Half of the architects mentioned the time limit: time was too
short to create a complete architecture for the case, utteringthat their models and documentation were unfinished by thetime the session ended. One architect rather liked the case,as it demonstrated the realistic scenario of a board that does
not accurately know what it wants. Overall the architects
found the RCC helpful in better structuring the architecture
documentation and the reasoning process.
V. A NALYSIS AND D ISCUSSION
Answering whether the RCC is an effective method tostimulate the rationale process, boils down into two questions
we need to analyze: ““are the different rankings in agreement
about the quality of the architectures?”, and “perform theparticipants in the test group better than the control group?
”. In the next subsections, we analyze the results to answer
these questions.
A. Agreement Between the Rankings
A scatter plot of the rankings is depicted in Figure 2. The
plot shows that each solution is either in the lower-left or inTable IV
A NALYSIS OF THE DIFFERENT SOLUTIONS , BASED ON THE CODING SCHEMA IN TABLE I

Word countRationale type
Design decision
Problem
Ident.
analysis
Descr.
Constraint
Ident.
analysis
Descr.
Asumption
Ident.
analysis
Descr.
OptionIdent.
analysis
Descr.
Benefit
Ident.
analysis
Descr.
Weakness
Ident.
analysis
Descr.
Trade-offIdent.
analysis
Descr.
Risk
Ident.
analysis
Descr.
Evaluation
Reflection
Total

C1
311

C2
251

C3
114

C4
317

C5
605

Total
1598

T1
289

T2
511

T3
284

T4
490

T5
472

Total
2046

2

3
1

2

2
3
2

4
4

9
8
6

4
3
2

8
4

3
3
4

1
4

4
2
1

20
16
7
5
1
5
0
20
12
18
4
4
4
1
1
1
1

5
1
5
2
1
5
1

2
1
6

2
4
3

2
2

6
5
4

14
13
18
1

3
6
1

10
2
6

3
1
3

3
1
7
3
4
4
1
1
1
1

1
2
1
1

11

13

11

11

23

69

19

32

17

42

12

the upper-right quadrant. Furthermore, the grayscale of thesolutions shows that solutions in the upper right quadrant are
higher ranked by the external architect as well.
The scatter plot indicates a high corration between the
different rankings. To confirm this, we calculated Spearman’srank correlations pairwise (Table V). As the results shown,
all three rankings have a very high, significant correlation.
From these correlations we may conclude that more rationale
is positively correlated with better quality. To exclude thepossibility that word count influences the rankings as well, we
used Spearman’s rank correlation to evaluate whether word
count has an influence fairly high correlation, At the tableshows, the correlations with word count are not significant.
B. Effectiveness of the RCC
Each of the rankings have the same solutions in the top 5,
although the exact order slightly differs between each of them.
Another observation is that in this top 5, four solutions comefrom participants of the test group. This is a clear indicator thatthe participants in the test group perform better than the control
group. To test this hypothesis, we used Wilcoxon’s signed rank
exact test on each of the rankings to evaluate whether the test
group performs better by comparing the median rank with thecontrol group. As Table VI shows, the significance of eachof the rankings is smaller than 0.05. We thus may conclude
that participants in the test group create solutions of better
quality than the control group.
Another aspect of effectiveness of the method concerns
the time dimension. Many practitioners consider lack of time
and/or budget (60.5%) as the most common cause of not
documenting design rationale [5]. In this study, all participants
had two hours to fulfill the exercise. Although on average,
the participants of the test group took more time (m=130),
than the control group (m=107.2), the difference was not

122

significant (U=5.500, p=.138). In other words, our proposed
method is non-invasive, and supports the process well, which
was confirmed in the feedback by the participants. Hence, we
may conclude that participants in the test group do not takemore time than participants in the control group to comewith solutions of better quality.
C. Influence of Rationale Techniques
As a next step, we compared the rationale techniques present
in the solutions with their ranking. For the total number ofrationale techniques applied, and for each individual rationale
technique, we created two hypotheses:
H0T There is no difference in the frequency of technique
T between the groups.
H1T There is a difference in the frequency of technique
T between the groups.
After analysis, it turns out that for none of the hypotheses
the difference is significant. Hence, the individual techniques
used do not account for the observed phenomena. Interestingly,
test groups use more rationale techniques: the frequency is
increased with nearly 77%. Also, the spread of which rationale
types occur are more comprehensive in the test group. The
participants of the test group used almost all rationale types(16 / 19 were present), whereas the control group omitted 12
rationale types (7/19 were present). It is interesting to note thatthe increase in coded rationale types seem to be widespread,and not due to isolated elements. Comparing the rationale
in the documentation of the top 5 solutions with the othersolutions, reveals that all solutions in the top 5 contain morerationale techniques than the others.
D. Threats to Validity
As for all experiments, there are threats to validity that need
to be addressed [18].

T4

1

T2

2

C5

Documentation freq. rank

3

Expert rank

T1

4

a 1
T3

5

a 3
a 5

C2

6

a 7
T5

7

a 9
C1

8

C4

9
10

C3
10

9

8

7

6

5

4

3

2

1

Total points rank
Figure 2. The rankings of Table II visualised, where the axes represent the total number of points attributed by the participants (horizontal) and the document
frequency rank (vertical). The external expert rank of is represented by the grayscale, where darker (black) is a lower rank, i.e., a higher perceived quality.
An approximate monotonic relationship can be observed between all three variables.
Table VS PEARMAN ’ S RANK CORRELATION FOR THE RANKINGS . VALUES MARKED WITH + HAVE SIGNIFICANCE p < 0, 05, VALUES MARKED WITH ∗ HAVESIGNIFICANCE p < 0.001, UNMARKED VALUES HAVE SIGNIFICANCE p > 0, 05.
Ranking
PointsDocumentation
External
Word count

Points1
ρ= 0.879 +ρ= 0.976 ∗ρ=-0.534

Documentation
ρ = 0.879 +1
ρ = 0.915 ∗ρ =-0.612a) Internal Validity: In this experiment, the participants
form the largest treat to internal validity. Familiarity is apotential influencing factor: some of the participants are morefamiliar with the research than others. All of the participatingarchitects were interviewed in an earlier stage of the researchon how they document design rationale. The experiment was
split in several sessions, as not all architects could participate
at the same time. Therefore, all participants were given thesame elaborate introduction to ensure all participants haveequal knowledge before going into the experiment. Still, eacharchitect is different and has different levels of knowledge
about the architecture tool used in the experiment. Therefore,

Table VI
W ILCOXON SIGNED RANK TEST. E LEMENTS MARKED WITH ∗∗ HAVESIGNIFICANCE p < 0.001.
Variable
PointsDocumentation
External

Test statistic Z
2.505
2.555
2.505

Exact p-value
0.006∗∗
0.004∗∗
0.006∗∗

External
ρ = 0.976 ∗ρ = 0.915 ∗1
ρ =-0.588

Word countρ= -0.534
ρ= -0.612
ρ= -0.588
1

we controlled as many factors as possible to create equalgroups.
An influencing factor with respect to the quality of the
rankings is the familiarity of the participants with each other.
As all architects have the same affiliation, it can well bepossible that architects know each others’ way of writing,style and output habits of the participants they have to rank.
As there is no way to control this, each architect had torank 3 solutions. Additionally, the ranking and integrity of thearchitects was constantly emphasized: architects are supposed
to rank solutions rather than individuals. This premise wasemphasized by the researchers both during the experiment andin the instructions for the ranking.
b) External Validity: All architects that participated inthis experiment work at the same organization. Furthermore,
a population of only 10 architects is limited. For that reason,
generalizing the results is difficult. Still, our findings point in
the same direction as other researchers found in the domainof Software Architecture [13]–[15], [25], that all pointed outthat some external stimulus such as checklists or prompting

can help designers to reason more, and that more reasoning
can have a positive impact on the quality [16].
c) Reliability: Rankings created by people are subjective
by nature. Therefore, we studied and compared multiple
rankings, and to which extent these rankings agree. The highcorrelations (¿.9) between the three independent variables
points, external expert ranking and documentation frequency
show reliability to be quite high.
VI. C ONCLUSIONS
Many practicioners acknowledge the need for documenting
design rationale, yet experience a lack of tool support [5].
Recent studies show that simple means can help in providing
more rationale, such as a cardgame [14], checklists [15], oran external observer [13]. However, architecture frameworks
and language have little to no support for capturing design
rationale.
In this paper, we studied the effect of design rationale toenterprise architecture documentation on design quality. We
propose the Rationale Capture Cycle to support architects intheir reasoning process. The idea of the method is that it isnon-intrusive, i.e., that it does not hamper architects in theirdesign process.
To measure the effectiveness of the proposed method, we
performed a controlled experiment. Ten experienced architects
participated by creating an architecture for a specificallydesigned case. Half of the architects received the method
as treatment, the other half did not. Through three different
measures we evaluated the design quality: partial rankings
based on points by the architects themselves, by coding
the rationale the architects documented, and by an externalarchitect.
The results of our experiment show a large agreement
between the different rankings. From the results, we may
conclude that solutions with more rationale are significantly
higher ranked in all three rankings, and that architects
using the method perform significantly better, confirming
our initial research question that the RCC is an effective
method. At the same time, the experiment shows that the participants do not take significantly more time to produce thearchitecture, countering the general believe that documenting
rationale is too expensive in time and/or budget [5].
Although the results look promising, the experiment showsthat not all architects fully utilized the RCC. As a consequence, the results are smaller than they could have been.
Only 2 architects made full use of the cycle, causing a large
difference in the comparison. Further research into how sucha model can be fully implemented during architecture design
is needed. We believe that with more research into actively
embedding the reasoning model in architecture frameworks,
further improvements can be made.

R EFERENCES[1] ISO/IEC/IEEE, “Standard 42010 - Systems and software engineering —Architecture description,” ISO/IEC/IEEE, Tech. Rep., 2011. [Online].
Available: http://www.iso-architecture.org/ieee-1471/[2] A. Jansen and J. Bosch, “Software architecture as a set of architectural design decisions,” in Working IEEE/IFIP Conference on Software
Architecture. IEEE, 2005, pp. 109–120.
[3] E. Poort, “RCDA: Risk- and cost-driven architecture: A solution architect’s handbook,” CGI Netherlands B.V., Tech. Rep., 2014.
[4] H. W. J. Rittel and M. M. Webber, “Dilemnas in a general theory ofplanning,” Policy Sci., vol. 4, pp. 155–168, 1973.
[5] A. Tang, M. A. Babar, I. Gorton, and J. Han, “A survey ofarchitecture design rationale,” Journal of Systems and Software,
vol. 79, no. 12, pp. 1792–1804, 2006. [Online]. Available:
http://dx.doi.org/10.1016/j.jss.2006.04.029[6] D. Perry and A. Wolf, “Foundations for the study of software architecture,” ACM SIGSOFT Sofware Engineering Notes, vol. 17, no. 4, pp.
40–52, 1992.
[7] J. Conklin and M. Begeman, “gIBIS: a hypertext tool for exploratory
policy discussion,” in Proceedings of the 1988 ACM conference on
Computer-supported cooperative work. ACM, 1988, pp. 140–152.
[8] A. MacLean, R. Young, V. Bellotti, and T. Moran, “Questions, options,and criteria: Elements of design space analysis,” Human-Computer
Interaction, vol. 6, no. 3-4, pp. 201–250, 1991.
[9] W. Regli, X. Hu, M. Atwood, and W. Sun, “A survey of design
rationale systems: Approaches, representations, capture and retrieval,”
Engineering with Computers, vol. 16, no. 3-4, pp. 209–235, 2000.
[10] J. Verries, A. Sahraoui, and M. Paludetto, “Design rationale in systemdesign,” in International Conference on Systems Engineering, 2008, pp.
380–385.
[11] T. O. Group, he Open Group Architecture Framework Version 9.1. The
Open Group, 2011.
[12] J. Leite, F. Oquendo, and T. Batista, “Sysadl: A sysml profile forsoftware architecture description,” in European Conference on Software
Architecture, ser. LNCS, vol. 7957. Springer, Berlin, 2013.
[13] M. Razavian, A. Tang, R. Capilla, and P. Lago, “In two minds: howreflections influence software design thinking,” Journal of Software:
Evolution and Process, vol. 28, pp. 394–426, 2016.
[14] C. Schriek, J. M. E. M. van der Werf, A. Tang, and F. Bex, “Softwarearchitecture design reasoning: A card game to help novice designers,” inEuropean Conference on Software Architecture, ser. LNCS, vol. 9839.
Springer, Berlin, 2016, pp. 22–38.
[15] M. Keil, L. Li, L. Mathiassen, and G. Zheng, “The influence of checklists
and roles on software practitioner risk perception and decision-making,”
Journal of Systems and Software, vol. 81, pp. 908–919, 2008.
[16] A. Tang, M. Tran, J. Han, and H. Vliet, “Design reasoning improves
software design quality,” in Quality of Software Architectures, ser.
LNCS, vol. 5581. Springer, Berlin, 2008, pp. 28–42.
[17] K. Peffers, T. Tuunanen, M. A. Rothenberger, and S. Chatterjee, “Adesign science research methodology for information systems research,”
Journal of management Information Systems, vol. 24, no. 3, pp. 45–77,
2007.
[18] C. Wohlin, P. Runeson, M. Höst, M. Ohlsson, B. Regnel, and A. Wesslén,Experimentation in Software Engineering. Springer, Berlin, 2012.
[19] R. Deneckère, C. Hug, J. Onderstal, and S. Brinkkemper, “Method
association approach: situational construction and evaluation of an
implementation method for software products,” in Research challenges
in information science. IEEE, 2015, pp. 274–285.
[20] A. Tang, “Software designers, are you biased?” in Workshop on SHAring
and Reusing Architectural Knowledge. ACM, 2011, pp. 1–8.
[21] L. Bass, P. Clements, and R. Kazman, Software Architecture in Practice.
Addison Wesley, Reading, MA, USA, 2012.
[22] R. McCall, Critical Conversations: Feedback as a Stimulus to Creativity
in Software Design. Springer, Berlin, 2013, pp. 11–40.
[23] J. Lee, “Design rationale systems: Understanding the issues,” IEEE
Expert. Syst. their Appl., vol. 12, pp. 78—-85, 1997.
[24] J. Burge and D. Brown, Design Rationale Types and Tools. AI in
Design Group, 1998.
[25] U. Heesch, P. Avgeriou, and A. Tang, “Does decision documentation
help junior designers rationalize their decisions? a comparative multiplecase study,” Journal of Systems and Software, vol. 86, pp. 1545––1565,
2013.