# Improving UX Work in Scrum Development: A Three-Year Follow-Up Study in a Company

Kati Kuusinen

► To cite this version:

Kati Kuusinen. Improving UX Work in Scrum Development: A Three-Year Follow-Up Study in a Company. 5th International Conference on Human-Centred Software Engineering (HCSE), Sep 2014, Paderborn, Germany. pp.259-266, 10.1007/978-3-662-44811-3_17. hal-01405083

HAL Id: hal-01405083

https://inria.hal.science/hal-01405083

Submitted on 29 Nov 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Improving UX Work in Scrum Development: A Three-Year Follow-Up Study in a Company

Kati Kuusinen

Tampere University of Technology, Tampere Finland

kati.kuusinen@tut.fi

**Abstract.** This paper presents a three-year follow-up study considering the improvement process of user experience work in a software company utilizing Scrum. Problems encountered in the organization included managing the product vision, timing of UX, lack of cooperation among disciplines, and understanding user needs. We also observed changes in the organization over two years. They included ceasing the centralized UX team and dividing the UX specialists over business lines. UX specialists were given influential roles in regard to product decisions – such as nominated as product owners.

**Keywords:** User experience (UX); Scrum, Agile development.

## 1. INTRODUCTION

Agile methods [5] are commonly utilized in industrial software development. However, guidance on those methods still lacks user experience (UX) [4] activities. Current recommendations for agile UX development suggest dividing the work into activities that are conducted within agile development and activities that are prior to development Sprints as design upfront work [1]. They also typically separate UX related work to its own stream to be conducted by separate UX specialists. However, organizations using such practices struggle with issues such as balancing the amount of design upfront work, and communication between developers and UX specialists [1]. Separating the UX designer to another stream seems to keep the UX designer out of the core team, which hinders within-project communication and endangers the realization of UX design as the product vision gets blurred [3].

In this paper we report a three-year follow-up study where a company advanced in Scrum<fnref n="1" /> and with an established UX team aimed at a better integration of those disciplines. We started the study in 2011 by a current state analysis to explore the situation of that time and to enable and focus improvements. We conducted an international web survey with 31 open- and 19 closed-ended questions, followed by

<footnote n="1">¹ Schwaber, K. Agile project management with Scrum, 1st ed., Microsoft Press. 2004.</footnote>
17 theme interviews [6]. Moreover, during 2012 and 2013, the organization adopted new ways of working and we measured their impact with surveys and interviews.

## 2. RESEARCH PROCESS

The studied company was mainly producing specialized software that was supplied via internet service providers (ISP) for consumers. The main product of the company was a software system with massive yearly releases. Large multinational ISPs were dominant when deciding of the feature content for the next release. The company had about 800 employees mainly in Finland, France, and Malaysia. The company was utilizing their own Scrum-based process model in their development. During 2011 the company had a centralized UX team with about 15 members and a few distributed UX specialists. In the beginning of 2013 the UX team as such was ceased and the UX specialists were distributed through the new business lines.

*Study conducted in year 2011.* We started the study in 2011 with a current state analysis to determine where and how to change working practices in order to improve the UX work in the company. Of the company employees working in development related roles from Finland, France, Malaysia and Russia, 76 responded to a web survey. Participant roles included developers, managers, architects, product owners, scrum masters, UX specialists, and quality engineers. The survey consisted of 50 questions (31 open- and 19 closed-ended) on processes and tools, collaboration and communication, and concepts and knowledge in the company [6]. Moreover, we conducted 17 interviews on UX specialists, product line managers, product owners, scrum masters, architects, developers, and quality engineers on the Helsinki site.

*Monitoring the organization performance in year 2012.* The cooperation between the researcher and the organization was smaller during 2012. The researcher mainly observed a new practice in the organization and discussed and gave some guidance to a few members of the organization as they implemented changes in the organization.

*Study conducted in year 2013.* We started the second current state analysis with a preliminary mapping consisting of five interviews in spring 2013. The interview data was enriched with unofficial "corridor discussions" and some email discussions. The aim of the preliminary mapping was both to gain a high-level understanding of the current state in the agile UX work and to create a preliminary list of possible changes to enable further ideation and discussion. We conducted the actual round of measurements inside one business line in October 2013 by surveying and interviewing six persons working for the business line at the Helsinki site of the company. First, all the participants filled in a short web survey individually (Table 1) and later we interviewed them in pairs. The participants represented the same business line from where the majority of the participants of the 2011 study were. We selected participants using theoretical sampling; with an approach that does not aim at representative sample but a sample that offers to compare to previous findings [2].

**Change** (question 6): as the business lines in the company had been rearranged recently, we asked respondents to compare the current situation within the business line to the situation that prevailed a year ago in the organization the respondent were
then working for. We use the abbreviations of the studied areas presented in Table 2 throughout this paper.

<table><caption>**Table 1. Survey questions, their types and used scales**</caption><tr><td>1.</td><td>The overall satisfaction of UX related work in the business line<br>Measured on a scale from 1 (not at all satisfied) to 7 (completely satisfied)</td></tr><tr><td>2.</td><td>Issues in UX related work that the respondent is currently most dissatisfied with<br>Open-ended question, listing of 0-3 issues</td></tr><tr><td>3.</td><td>Issues in UX related work that the respondent is currently most satisfied with<br>Open-ended question, listing of 0-3 issues</td></tr><tr><td>4.</td><td>Means to improve the current situation<br>Open-ended question, text area</td></tr><tr><td>5.</td><td>Level of performance on certain areas of UX related work and their importance to project success. The studied areas (11 items) are listed in Table 2.<br>Fourfold table: x-axis: performance from poor to excellent, y-axis: importance from insignificant to significant.</td></tr><tr><td>6.</td><td>The change within last year in the level of performance on the studied areas (areas are listed in Table 2)<br>Change was evaluated on the following scale: worsened greatly, worsened somewhat, worsened slightly, no change, improved slightly, improved somewhat, improved greatly</td></tr></table>

<table><caption>**Table 2. Studied areas of agile UX work. Level of performance and importance for project success and experienced change were measured for these areas (see items 5 and 6 in Table 1).**</caption><thead><tr><td>Abbreviation</td><td>Studied area</td></tr></thead><tbody><tr><td>A</td><td><strong>Agility</strong> of UX work</td></tr><tr><td>T</td><td><strong>Timing</strong> of UX work</td></tr><tr><td>W</td><td><strong>Welcoming</strong> late change</td></tr><tr><td>C</td><td><strong>Competence</strong> in the project team</td></tr><tr><td>B</td><td>Maintaining the <strong>big picture</strong> of the project</td></tr><tr><td>U</td><td><strong>Understanding</strong> user needs</td></tr><tr><td>M</td><td><strong>Meeting</strong> user needs</td></tr><tr><td>F</td><td>Getting user <strong>feedback</strong></td></tr><tr><td>P</td><td>Cooperation between <strong>product owners</strong> (PO) and UX specialists (UXS)</td></tr><tr><td>D</td><td>Cooperation between <strong>developers</strong> and UX specialists</td></tr><tr><td>Q</td><td>UX implementation <strong>quality</strong></td></tr></tbody></table>

### 3. RESULTS

Next, we present the results from our study in two parts. First, we briefly revisit findings from the study conducted during 2011 already reported in [6]. Then, we concentrate on the results of the study conducted in 2013, and address the changes that can be observed.
## 3.1 Year 2011 – Analyzing the problems in agile UX work

The company had had some issues in integrating UX work into other agile development practices. They wanted to clarify their understanding of the problems in order to improve the situation. In a web survey answered by 76 R&D related employees and with 17 interviewees we determined the following: 1) Which issues are considered as the biggest challenges in the agile UX work; 2) Which tasks are considered as the most important duties of the UX team; and 3) How the employees would improve the agile UX work and communication between different R&D roles.

The top three challenges in agile UX work included managing the big picture of the project, changing the UX team's way of working from traditional waterfall practices to agile practices, and proper timing of UX work in agile process. Other major challenges were lack of cooperation between different roles, and being able to understand user needs and fulfilling them by the software (Figure 1).

<figure><img src="image_2.png" /><figcaption>Figure 1. Biggest challenges in agile UX work by respondent role (results 2011). Horizontal axis presents the amount of respondents. Legend: QA = quality assurance, PO/SM = product owner/ scrum master, UX = UX specialist, Mgmt = manager, Dev = developer. Total number of respondents was 60; each named 1-3 issues.</figcaption></figure>

Study participants emphasized the most that they would like to have closer collaboration between UX specialists and other development roles. The most often mentioned manner of improving the cooperation was to include UX specialists in Scrum teams instead of having them work in a separate UX team.

## 3.2 Year 2012 – Implementing new practices in R&D

The company implemented a method called “UX review board” to raise the visibility and understanding of UX related problems of products [6]. The aim was to get UX issues into backlog items to ensure and fasten the process of actually fixing those.
The practice improved the visibility and understanding of UX issues. However, it did not work as planned – in many cases the product owner agreed the found issues are severe but still did not welcome them in the backlog as the PO felt the backlog was already overloaded with other issues. Singh [7] reported similar problems; priority of UX tasks remained low and thus they rarely got implemented.

The company adopted a new role of UX engineer to the R&D organization. The role was to work inside Scrum teams as a link between UX designers and developers. UX engineers were people with both coding and UX skills. First, the company had difficulties in recruiting such multi-talents. Later, they noticed they did not manage to maintain the work contents of the role as planned; UX engineers were observed to mainly concentrate on implementing the UI. It became evident that it required changes on the organization level to improve the impact of the agile UX work; the organization structure seemed to hinder from improving the effectiveness of the UX work.

## 3.3 Year 2013 – Changing the organization structure

There were plans to make noticeable changes on the organization level as the previous attempts had not had a desired effect. We started a second round of current state analysis in the beginning of 2013. We used a more lightweight approach than previously to save time and costs. We started by preliminary theme interviews, continued with an expert evaluation and group discussion, and finally conducted a web survey and structured pair interviews. Next, we present the results.

*Background mapping.* The idea of preliminary interviews and discussions was to gain understanding on how employees would like to change the current state and which issues they consider the major challenges in agile UX work at the moment. A researcher interviewed five persons from the organization and based on the findings formed a suggestion on how to improve the current state of agile UX work. The main items of the suggestion were the following:

* "Away from silos"; UX needs to be considered in every phase from roadmap to delivery and it must be everybody's concern, and team performance metrics should include a UX-related instrument.

* Mutual project goals, support goal-setting; developers still need more understanding of UX specialists' work contents and goals.

* Spread UX mindset in the organization; continuing the "cultural change"; increasing the understanding of the importance and meaning of UX work.

* Prototype and utilize multi-phase testing on actual users.

* Emphasize early cooperation and iterative way of working.

*Expert evaluation: Comparing the situations of years 2011 and 2013.* During summer 2013, an experienced UX specialist walked through the report of the study conducted in 2011 and commented on how they thought the situation had changed since. Next, we list the found issues and the comments of the UX specialist. Found issues are numbered and comments of the UX specialist are started with "UXS:" and printed in italics.
1. UX work is not really integrated with R&D ways of working. (2011)

- UXS: "To the best of my knowledge, this is not true anymore. **All the UX team's work I see is done in close collaboration with developers.**" (2013)

2. It seems that the importance of UX work is generally acknowledged but there is a wish that it should not cost or take time. (2011)

- UXS: "*This, I think has improved ... projects try find special sprints to clean up UI ... or there really is no work on UI that is not involving the UX team.*" (2013)

3. UX issues are not ordered together with other issues in backlog. Lack of considering UX when creating the minimum viable product.

- UXS: "*Both issues are (slowly) improving.*" (2013)

4. Inadequate communication between architects and the UX team. (2011)

- UXS: "*This has improved greatly. Architecture is one of the main issues affecting the UX as experienced by consumers.*" (based on experiences in development of two separate products) (2013)

5. UX work is often conducted in reactive manner. The UX team needs to hurry to get ahead of implementation, but they report often failing in doing so. (2011)

- UXS: "*In my experience UX team(s) are quite busy. ... In the projects I have been involved in the design has not been reactive. ... The work is in any case creative collaboration - so agile.*" (2013)

6. UX work is not properly integrated with the R&D work. (2011)

- UXS: "*To some extent I do feel UX will always be a little outside of development; it is a different mindset with different educational background and as a discipline it is UX team's role to challenge.*" (2013)

## 3.4 Measurement of the situation after the organizational level change

The organization was rearranged and the centralized UX team as such was ceased in fall 2013. UX specialists were divided to different product lines, and they were given more influential positions in the organization.

*Issues in UX related work that respondents were most satisfied with.* We asked the participants to list issues that they are most satisfied with in the current ways of working. In general, participants reported that the organization has positive and active attitude; work gets done, tasks are novel and interesting, and new technologies and tools are adopted. Both developers' and UX specialists' skills in UX and attitude towards it were appreciated. UX specialists' cooperation with developers, product management, and marketing were all mentioned. Also, user participation both in planning and feedback gathering was considered good.

Nominating an experienced UX specialist as a product owner had given the person more influence on the product vision. However, it was criticized that it was more of a career decision than actual change on the process level: if the person e.g. chooses to leave the company, it is not a company policy to nominate another UX aware person to replace them.
Issues in UX related work that respondents were most dissatisfied with, and means to improve the current situation. Diversity and poor quality of used tools caused dissatisfaction. Both own and several third party UI frameworks were in use which was reported to cause extra work. In addition, delivering several clients instead of building one product with several touchpoints was criticized. Also, it was mentioned that the current development tool did not allow creating high-quality UX.

There was still a small number of UX specialists in the company compared to the amount of UX work. UX specialists were overwhelmed with work and that led to implementing new features without UX design, or with "very basic design" that was mentioned to be often late. Also, both an architect and a UX specialist reported that the visibility of the contents of concept and design work and timing of those tasks should be improved: "...the implementation request often comes as a surprise, or I find out after the implementation has begun" - an architect.

*Level of performance on studied areas and their importance to project success.*
We asked the participants to evaluate both the importance of the studied areas to project success (from unimportant to important) and the performance of their organization in those areas (from poor to excellent) on a fourfold table (Figure 2). The most important areas for project success were maintaining the big picture of the project (76.7), understanding user needs (75.0), project team competence (74.5), and proper timing of UX work (71.7). The organization performed best on maintaining the big picture (56.2), project team competence (55.2) and getting user feedback (53.3).

<figure><img src="image_3.png" /><figcaption>Figure 2. Averages of performance and importance (middle and rightmost columns) on certain areas (letters T-W) and their standard deviations arranged by the difference between the importance and performance (leftmost bars without standard deviation lines). T = Timing of UX work, U = Understanding user needs, P = PO-UXS cooperation, A = Agility of UX work, D = Developer-UXS cooperation, Q = Quality of UX implementation, B = Big picture maintaining, M = Meeting user needs, C = Competence in project team, F = Getting user feedback, and W = Welcoming late change.</figcaption></figure>
There were largest gaps between importance and performance in the following: timing of UX work (29.0), understanding user needs (26.8) and cooperation between UX specialists and product owners (26.0). Participants agreed the most on their level of performance in big picture management (SD=9.6), and welcoming late change (SD=10.3). Largest values of standard deviation were measured for timing of UX work (27.0) and cooperation between developers and UX specialists (25.3). Both the lowest and highest assessments of performance were given by a developer and an architect. It probably indicates that there are project or individual-specific ways of working inside the business line.

## 4. SUMMARY AND CONCLUSIONS

This paper presents a three-year follow-up study considering the improvement process of agile user experience work in a software company. In the initial state the organization was following Scrum methodology in software development and it had an established UX team. However, the organization had problems with integrating the work of these two disciplines. The major problems included managing the big picture of the project (product vision), timing of UX work, lack of cooperation between disciplines, and understanding and fulfilling user needs. We conducted major changes in the organization over two years. Changes included ceasing the centralized UX team as such and dividing the UX specialists over different business lines. UX specialists were given more influential roles in regards to product level decisions. These practices were reported to improve the big picture management and understanding user needs. However, there were significant differences in evaluations between individuals in timing of UX work, cooperation between UX specialists and other disciplines, and the agility of UX work. This indicates that the practices still are strongly person-dependent and there is need for more established approaches.

## 5. REFERENCES

1. da Silva, T., Martin, A., Maurer, F. and Silveira, M. User-centered design and Agile methods: a systematic review. *Proc. of the International Conference on Agile Methods in Software Development (Agile 2011).*

2. Eisenhardt, K. M. Building theories from case study research. Academy of Management Review, Oct 1989; 14, 4, 532-550

3. Hodgetts, P. Experiences integrating sophisticated UX design into agile process. *Proc. AGILE 2005 Conference*

4. ISO 9241-210:2010. Ergonomics of human-system interaction. Part 210: Human-centered design for interactive systems

5. Highsmith, J., & Cockburn, A. (2001). Agile software development: The business of innovation. Computer, 34 (9), pp.120–127

6. Kuusinen, K., Mikkonen, T. and Pakarinen, S. Agile user experience development in a large software organization: good expertise but limited impact. Human-Centered Software Engineering. Springer Berlin Heidelberg, 2012. 94-111.

7. Singh, M. U-SCRUM: An Agile Methodology for Promoting Usability. Proc. of AGILE '08. 2008 pp. 555-560. Toronto, Canada, Aug 4-8, 2008. IEEE Computer Society.