Measuring teamwork quality in large‐scale agile organisations evaluation and integration of the aTWQ approach

IET SoftwareVolume 15, Issue 6 p. 443-452 ORIGINAL RESEARCH PAPEROpen Access Measuring teamwork quality in large-scale agile organisations evaluation and integration of the aTWQ approach Alexander Poth, Corresponding Author Alexander Poth alexander.poth@volkswagen.de orcid.org/0000-0002-2868-5633 Volkswagen AG, Wolfsburg, Germany Correspondence Alexander Poth, Volkswagen AG, Berliner Ring 2, D-38436 Wolfsburg, Germany. Email: alexander.poth@volkswagen.deSearch for more papers by this authorMario Kottke, Mario Kottke Volkswagen AG, Wolfsburg, GermanySearch for more papers by this authorAndreas Riel, Andreas Riel Grenoble INP, G-SCOP, CNRS, Grenoble Alps University, Grenoble, FranceSearch for more papers by this author Alexander Poth, Corresponding Author Alexander Poth alexander.poth@volkswagen.de orcid.org/0000-0002-2868-5633 Volkswagen AG, Wolfsburg, Germany Correspondence Alexander Poth, Volkswagen AG, Berliner Ring 2, D-38436 Wolfsburg, Germany. Email: alexander.poth@volkswagen.deSearch for more papers by this authorMario Kottke, Mario Kottke Volkswagen AG, Wolfsburg, GermanySearch for more papers by this authorAndreas Riel, Andreas Riel Grenoble INP, G-SCOP, CNRS, Grenoble Alps University, Grenoble, FranceSearch for more papers by this author First published: 12 August 2021 https://doi.org/10.1049/sfw2.12036Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract The maturity of organisations is measured using process assessment models like the International Organization for Standardization (ISO)/IEC 33001. Product quality is refined with internal and external product quality characteristics based on models like the ISO/IEC 25010. With the shift from the Tailorism-driven process orientation to a more people-centric organization, the process and product quality dimensions have to be extended by the teamwork quality (TWQ) dimension. For the sake of systematic organization development, this new third quality dimension needs to be integrated into existing quality frameworks to achieve a holistic quality approach. This article introduces agile TWQ as the method of choice. It explains its design, implementation and evaluation in the large enterprise context of the Volkswagen AG. It provides insights into the effectivity of this approach in heterogeneous multi-team settings, as well as into its strengths, weaknesses, and limitations. 1 INTRODUCTION AND MOTIVATION In order to adapt to the ever faster evolving challenges linked to the economy, ecology, as well as the social domain, numerous companies in various sectors see themselves confronted with the need for agile transformation [1-3]. The latter may affect the entire organization at a large-scale, at the scale of multi-teams, as well as of individual teams. The transformation can be driven either top-down or bottom-up, as well as both together at the same time [4]. With emphasis of the agile working paradigm, the focus on the transition process becomes a cultural change towards an agile mindset. Figure 1 shows the key role of the teams within an agile organization. The team creates and evolves the products and services of an organization. The governance has to establish procedures to guide and facilitate the teams' daily business. Furthermore, the governance has to assess the delivered products and services to ensure compliance to market and domain regulations and standards. The agile project review supports these tasks, which can be scaled from individual project teams to bigger organizational entities [5]. FIGURE 1Open in figure viewerPowerPoint Interaction of the team with deliverables and the enterprise governance Established standards for product quality like the International Organization for Standardization (ISO) 25010 [6] for software, and the universal ISO 9000 [7] lack focus on teams. Instead, they scope on processes, which are not at the heart of agile organisations. A logical consequence of this is that traditional quality management (QM) has a focus on the product and process quality rather than on teamwork quality (TWQ). Other common approaches like Total Quality Management have some focus on human resource management and trainings for successful implementation [8]. However, they do not emphasize teams. Teamwork quality is the blind spot of an organization's success. This is where the 3-pillar quality approach comes in. It stresses the explicit demand for a new perspective on QM and evaluation. The product quality pillar represents the traditional and therefore most established QM domain. Its focus is on the outcomes and deliverables and their internal and external characteristics. With this product quality mindset, any product version has to be assessed to confirm the expected quality characteristics. This can be a time- and effort-intensive work whenever many complex products are delivered in highly frequent releases. With Tailorism, the process view was added as the second pillar of QM. The focus of process quality is to ensure high process determinism. The assumption is that the deterministic process activities result in deterministic process outcomes. The expectation is that higher process quality leads to higher outcome quality that is also reproducible. Over the years, different process quality standards and assessment approaches have been developed. This kind of process quality work is an indirect effort that should affect the product quality. The benefit behind this mindset is that not every product will have to be assessed if the applied process is validated as adequate for a specific product development. This scales down efforts for dedicated product quality assessments by assessing only the product development process. The additional third pillar is team quality. The team quality focuses on the systematic enablement of the product team to do all relevant things needed to deliver a specific product within adequate quality. This includes that the teams can adopt their development and delivery procedures as well as the product itself for a better market fit. To do this, the teams need product- and domain-specific knowledge. This knowledge includes the technology and regulation requirements to define and design adequate products and related procedures. The benefit of the team-oriented mindset is that it reduces the amount/the complexity of process and procedure assessments by assessing only the product team. This usually requires a significant amount of time and effort in case of complex development environments with frequent adaptions of the delivery procedures. Especially for products that need to scale along the product life cycle, there is a demand for continuous adoption of new technologies, as well as the adaptation of the delivery flows. Figure 2 shows the three pillars of quality in agile organisations, starting from the uniqueness with the product quality approach via the repeatability with the process quality approach to the facilitation of the ability to create ('createability') with the team quality approach. The createability is achieved by the product and domain mastery of the team. Mastery leads to a high autonomy in terms of product and process quality aspects. The grey bar indicates the movement from the product via the process to the team. This also indicates that quality awareness is built and driven typically from left to right. It shows an abstraction from the product to more indirect quality drivers. FIGURE 2Open in figure viewerPowerPoint The three pillars of quality in agile organisations and their relationships The challenge is to assess the mastery of a team in a specific product domain context. Although any agile method emphasises the central role of teams and their abilities to collaborate, they lack systematic evaluation measures for rating TWQ. Certainly, such measures need to be aligned with the agile mindset in terms of team autonomy, lose governance coupling, holacracy [9], and similar. Therefore, classical top-down organization evaluation frameworks are likely to fail in a competing value framework [10]. Instead, lean methods that can neatly be integrated into the teams' regular agile activities applied in the form of a self-driven self-service approach have a higher chance to be accepted. To evaluate the team quality, there exist approaches like [11], which can be applied to every team. Established agile frameworks like SAFe® claim through the core values [12] terms like built-in quality [13] that shall address any quality relevant aspects. However, a deeper investigation of these frameworks reveals that they lack a clear and holistic quality practice set. A similar handling of the quality term can be found in Scrum. It focuses primarily on product quality aspects through the DoD [14]. To bring the generic quality term of agile frameworks and the elaborated teamwork approaches together, the agile Teamwork Quality (aTWQ) approach is proposed in [15]. Pursuing the objective to establish the third pillar of the concept in Figure 2, the research question derived from this introduction is as follows: How to establish an aTWQ approach that is applicable in different agile frameworks like Scrum or SAFe®? To answer this research question, an internal 'market analysis' of TWQ approaches is needed. The results provide the basis for the development of the aTWQ approach. During development, the relations to established agile frameworks have to be ensured. For rollout and establishment, an evaluation is required to ensure that the designed artefacts of aTWQ actually cover the demand. With the described approach, the gap in current TWQ approaches will be identified and closed with a focus on delivering a practicable approach whose industrial applicability will also be demonstrated. Section 2 presents related work in terms of a kind of 'market analysis'. Section 3 focuses on the development and enhancement of aTWQ. In section 4, the evaluation and improvement are shown and discussed. The last section summarises the article's contributions. 2 RELATED WORK This section discusses the reviews of existing works in the areas of TWQ in agile, as well as teamwork evaluation methods and approaches. 2.1 TWQ in agile Literature related to TWQ in general has been published for 2 decades [16]. For more than 1 decade, the topic has been investigated in the context of agile teamwork [17]. The team setup is an important part of organisations by allocating the right skills in agile teams for high performance [18]. Team maturity and agile methods focus on performance and present correlations [19]. In [20], it is shown that teamwork correlates with productiveness of agile software development teams. Also, TWQ and team performance are analysed in [21] with the observation that they impact the product quality positively especially in smaller teams. Reference [22] elaborates on correlations which are confirmed by team roles regarding the topic aTWQ. Another study [23] evaluates the relationship of personality and teamwork in agile working environments with the Bayesian model approach. In reference [24], gender-specific aspects on TWQ are analysed in the context of agile development teams. The work that it is specific per country and/or innovation domain presents an extension to TWQ to represent observations in creative research for sustainable innovation within teams [25]. The identified works do not define a systematic generically applicable approach for agile teams by linking and using artefacts of established agile frameworks as indicators for the facilitation of (self-) assessment. 2.2 Teamwork evaluation methods and approaches Teamwork aspects have been treated in literature, for example, in the context of assessment of teamwork skills in computer engineering and science education [26]. In [27], teamwork is assessed in an educational context. Teamwork and engagement in the context of feedbacks is analysed in [28]. Some of this previous work addresses aTWQ explicitly like in agile contexts [19] or innovation teams [16], some also propose organizational models fostering TWQ like [29] with the spiral dynamics model, which distinguishes team maturity characteristics. In [30], a systematic questionnaire is developed to determine TWQ. The three most promising structured teamwork approaches are the TWQ [16], which focus on quality indicators of teamwork. The Team Climate Inventory (TCI) [31], which has been developed and established over years, evaluates team indicators related to the teams' working structures for innovation. The third is the Group Development Questionnaire (GDQ) [32], which addresses both team development and maturity. The investigated works propose differently sized generic questionnaires, which are typically used by experts to assess teams. These questionnaires need to be consolidated and rephrased to also suit non-experts. This is a missing step for deploying them widely in autonomous agile teams. To summarise, existing works do not offer a ready to use approach to address the expectations of autonomous teams to elaborate their TWQ in a self-service way for continuous improvement. This leads to the need to bundle existing elements and building blocks to a holistic approach that is usable in industry. 3 CONTEXTUALIZING aTWQ TO FOSTER aTWQ In [33], based on [16, 34], we derived the initial team-level approach including the six aspects communication, coordination, balance of contribution, mutual support, effort, and cohesion. These six quality aspects also lead to team performance [11]. This legitimates economically the effort required for the actions of measurement and further TWQ improvement. The TCI with its focus on four key aspects and its short set of 19 questions extends the six key aspects covered by TWQ to a holistic team evaluation questionnaire for aTWQ. The objective of the questionnaire, as presented in [33], has been to create a concise single questionnaire by merging TCI and TWQ aspects. To achieve this, the TCI questions were extended by the missing TWQ aspects. Figure 3 shows the mapping of the questionnaire's main topics to Scrum and SAFe®, respectively, based on the specific approach's elements covering the aspects addressed by the topics. Hence, the TCI/TWQ questions represent generic practices, while the associated elements from Scrum or SAFe® represent specific practices of either approach. Both combined, constitute the practice set of aTWQ in a specific team environment. The indicators of the approaches are based on the current versions of SAFe® 5.1 [35]. For Scrum-based teams, they were built on the Scrum Guide version of November 2017 [36] and the version of November 2020 [14] to support the 'upgrading' and newly established teams. FIGURE 3Open in figure viewerPowerPoint Visualization of the related indicators of Scrum and SAFe® to agile Teamwork Quality (aTWQ) For the integration into the project reviews [37] to evaluate individual product teams, a group of teams (like programs), or entire organizational units, an extension beyond a typical team size is needed. In the context of aTWQ, people who have common goals within a purpose constitute a team. The team size is aligned with the agile definition of seven to nine individuals [38]. A group is a collection of people or teams coordinating outcomes and efforts. In the aTWQ approach, the extension to groups larger than one team is realized with the GDQ, since agile approaches that scale do not have 'one big team'. In SAFe®, for example, different types of teams exist, like the technical and business teams sharing a common basic approach. 'Both types of teams strive for fast learning by performing work in small batches, assessing the results, and adjusting accordingly' [39]. This leads us to deriving that in SAFe®, a group of different types of teams is managed. To handle this appropriately, something beyond TWQ is needed to show that the group forming an SAFe® environment works fine. Furthermore, it cannot be assured by default that teams of the same type share a common purpose or challenging goals within an Agile Releases Train (ART). Perfectly independent feature teams share only the same timeline for their deliveries during the ART. For the mapping, aTWQ uses the following observation of [40]: 'How are team climate theory (measured by TCI) and the integrated model of group development (measured by the GDQ) connected? … the two models are closely related'. The evaluation of the readiness of organisations is based on the spiral dynamics approach, which is usable in larger social systems like the GDQ. These two models provide the basis for using the aTWQ approach from individual teams to larger organizational units including many teams that work for some shared objectives. Based on the ISO/IEC 33001:2015 [41], a maturity rating of each question can be made in the four categories: No (0%–15%), Partly (16%–50%), Largely (51%–85%) and Fully (86%–100%). If needed, each topic can be evaluated based on the rating of its questions. The rating is needed at least in the context of the externally driven application of aTWQ, for the purpose of for example, comparing different teams. Figure 3 presents the relations of aTWQ v1.1 with Scrum and SAFe® (solid blue lines). The dotted red lines are new relations in version 1.1. The focus on Coordination of SAFe® in aTWQ v1.1 is more transparent. In the version 1.0, the designer avoided the high amount of relations with the topic Coordination. However, practice shows that especially in aTWQ self-service application, these missing relations can result in misunderstanding parts of the questionnaire. The relation links are set to show 'contributions' of rituals, artefacts etc. of the agile frameworks to aTWQ questionnaire. However, these contributions are indicators and do not assure that, for example, by fulfilling formal rituals (e.g. with Scrum theatre), the aTWQ topic is addressed adequately. Also, the links are not set in cases of weak indication/contribution to the aTWQ topic. The detailed questionnaire related to the topics is presented in [33]. 4 VALIDATION AND IMPROVEMENT OF aTWQ Within the Volkswagen AG, the evaluation, rollout and establishment of aTWQ have been introduced as follows: - Simulation: based on the historical experience of coached agile teams, the aTWQ approach is simulated. - Proof of Concept (PoC): first real team application of aTWQ under expert observation is conducted. - Feedback, Self-Service Kit (SSK) and Coaching: Rollout preparation with feedback loops and the creation of different service offers for team individual 'service consumption'. - Integration into project reviews: integration in governance procedures of the aTWQ as a step of establishment. - Continuous improvement: based on the collected observation information and first re-checks of teams with aTWQ, the establishment of a continuous improvement wheel starts. - Demand for technology: to ensure that tech teams are able to perform their specific product/service delivery/offer tasks adequately, the presented aTWQ needs a technology-specific extension. - Integration into other initiatives: for sustainable establishment, aTWQ is integrated into related initiatives. We will elaborate on each of these in the following subsections. 4.1 Simulation In the first step, the initially designed approach was simulated with the coaches of the Agile Center of Excellence (ACE) [3] and quality experts from the Quality Innovation Network (QiNET) [42]. The simulation was realized by virtual application of the aTWQ questionnaire to teams coached in the past. For each simulation, a specific point of time in the past was used as the timestamp for answering the aTWQ questions based on the team's situation at that time. During the simulation, the answers of the teams were simulated by the coaches/experts based on their knowledge about the team. Based on the answers, potential chances and risks for the team development were derived. Then the timestamp was moved forward to check if the chances or risks identified by the aTWQ approach were realistic to validate the questionnaire as a starting point for team improvements. 4.2 Proof of Concept An initial PoC was done in the Scrumban-aligned product team of Testing as a Service (TaaS) [43]. Two additional teams also helped to improve the questionnaire and the SSK iteratively. The self-assessments took approximately 1.5 h. Some facts about the TaaS PoC: The concerned service was introduced in 2016 and has been offered in the Volkswagen Group since 2017. Over the years, evolving team constellations have led to an established devops team with end-to-end responsibility for the service delivery. The team members' experience levels covered a wide range from junior developer to senior engineer. Recently, the team constellation was changed putting the team in a re-balancing phase. The application of the aTWQ questionnaire worked fine and was conducted as a dedicated task in a team retrospective. The identified enhancement potentials were used like retrospective outcomes and led to actions for team improvement. Some small improvements based on the feedbacks and observations were made which have already been integrated into Figure 3. 4.3 Feedback, SSK and coaching In the second step, two other applications were conducted in similar team sizes and the self-assessment was realized in a similar way to get more feedback from application and see how the self-service maturity is. A further reflection was provided by the Agile Community (AC) of the Volkswagen AG to get a more holistic feedback about the understanding of the questions from different people to see what else is needed as the explanatory and supporting material in an SSK. To support the scaling approach of the Volkswagen Group IT [5], an SSK has been designed with the questionnaire and additional information about 'how to use' the kit to establish a continuous improvement of the TWQ. Figures 1 and 2 are part of the SSK. For the latter, a demand for multi-language questionnaires was identified to avoid translation mis-interpretations. For teams with more support demand, the ACE provides coaching packages for the adoption of the aTWQ approach. 4.4 Integration into agile project reviews The integration into Agile Project Review [44] was the next step for the sustainable establishment of the aTWQ approach at the Volkswagen Group IT. The chosen integration approach was to conduct an aTWQ self-assessment in the teams. This makes it possible for each team to reflect and answer the aTWQ questions in their safe and private environments. Optionally, however, they may ask for facilitation by team externals, for example, an ACE coach. The self-assessment result will be used as an input to the agile project review in order to ensure that the team's individual interpretation of the questions is in a comparable range between different projects within the organization. The reviewers use the aTWQ answers to crosscheck them with the observations during the project review. As long as the answers and the observations are aligned, there will be no further discussion about the aTWQ during the agile project review. In case of deviations, the reviewers will try to get more information about the observed gaps. To make this new step and enhancement of the established project review official, the aTWQ approach needs commitment by the ACE and AC. To achieve this commitment, the setup and integration of the aTWQ in the project review will be discussed and feedback collected. The commitment to the extended agile project review was published in the AC in summer 2020. Agile project reviews initiated after this announcement will have to conduct the self-assessment before the agile project review. Currently, the observation is that the implemented frontloaded aTWQ self-assessment in the context of the agile project review works fine. Sometimes projects are late with their self-assessment result delivery, and the reviewers have to cross-validate the observation of the agile project review and the self-assessment results after the review session. However, as long as no significant gaps are identified, the review does not need to be re-conducted. In any case, gaps present a risk of re-work efforts, and therefore have to be minimised whenever this is possible. Another observation is that especially in programs involving several teams, the selection of the right aTWQ participation is difficult for the teams. Currently, the lessons learned are handled by the reviewer. In case of programme reviews, a short additional introduction into the aTWQ is given. During this introduction, the programme members are informed about how to set up the right aTWQ self-assessment. The current recommendation for starting with aTWQ is to select at least one team of each programme hierarchy level. If the programme decides to use aTWQ more intensively in their continuous improvement and/or agile transition, the rollout to other teams can be easily initiated. With this approach, the aTWQ is enforced at least as a trial in the projects and programs. However, teams can decide on their own if they want to adopt aTWQ in their strategic improvement and team evolvement work, depending on the benefit and prioritisation of aTWQ in their product roadmap. The integration into the project review is a way to gather feedback. For example, the aTWQ questionnaire version 1.1 added to the question 'Do people in this team always search for fresh, new ways of looking at problems'? to the Scrum practices 'refinement'. Some teams reflected the question in version 1.0 only regarding their working approaches and not regarding the products. But the question is open and both aspects need to be dealt with. An insight for the authors was that product teams handle their contractors and suppliers differently based on their established working approaches. Some product teams see them as a part of the delivery team, but others do not. Depending on the view, the scope of the conducted aTWQ assessment is completely different. Also, this leads to different application approaches. For example, the German Act on Temporary Employment Businesses demands an evaluation on the Original equipment manufacturer side and another on the supplier side, which makes some collaboration settings in some particular working approaches difficult. 4.5 Continuous improvement scenario In 2021, the first wave of the continuous improvement of teams with aTWQ re-assessments started. One of the findings so far is that it is useful to wait approximately 6–9 months before doing the second self-assessment. This gives the teams time to work on the actions derived from the initial aTWQ assessment. Work on these actions requires more than just implementing those. Teams need time to integrate the newly implemented actions in their team culture and mindset. This is the reason why the re-assessment should not be too close to the initial assessment. On the other hand, a re-assessment should not be conducted too late to avoid losing speed of the team's work quality improvement. Two of the observed teams had smaller changes in their team composition (one team substituted one person, while the other team integrated one additional person), in another one, roughly half the people changed (three changes and one additional person). Depending on the team and its individuals, the changes have more or less impact on the team's self-assessed maturity. In all cases, aTWQ helped to make the current maturity state transparent to the teams. Based on the state, actions for improvement could be derived to ensure and improve the teams' capabilities and efficiencies. We could not find any indication that after a team setup change, the concerned team should conduct an aTWQ self-assessment right after the change. We currently recommend to schedule a re-assessment (re-check) not before the new individuals have gone through the team integration phase. Another interesting, however still open, aspect is, how fast and how much a team setup needs to change to impact the aTWQ assessment outcome significantly. This is a product team's individual tipping point, which depends on many factors, not only on the addressed aspects and indicators of the aTWQ approach. However, the aTWQ approach can help to make the status quo transparent. This can reveal the probability of an upcoming tipping point (as it is the case) when too many product-relevant ratings are decreasing. Figure 4 shows the process where aTWQ should be used to identify significant issues. We found that a good practice is to evaluate the first data at the end of the transition phase, and then regularly every 6–9 months. FIGURE 4Open in figure viewerPowerPoint Best time to perform agile Teamwork Quality (aTWQ) 4.6 Demand for technology and domain regulations To address the product domain and technology-related requirements, the generic aTWQ approach can be refined. In one project, the authors extended the self-assessment with a technology-specific questionnaire. This questionnaire focusses on identifying the status quo of the team's maturity to handle the technology adequately. The objective is to avoid product risks by achieving the team maturity required to work and develop products with the technology. In that project, the cloud technology was assessed. In another project, a questio