Organisational climate assessments of Agile teams – a qualitative multiple case study

IET SoftwareVolume 14, Issue 7 p. 861-870 Case StudyOpen Access Organisational climate assessments of Agile teams – a qualitative multiple case study Eliezer Dutra, Corresponding Author Eliezer Dutra eliezer.goncalves@uniriotec.br PPGI/UNIRIO, Federal University of Rio de Janeiro State – Rio de Janeiro, RJ, Brazil CEFET/RJ, Celso Suckow da Fonseca Federal Center for Technological Education – Nova Friburgo, RJ, BrazilSearch for more papers by this authorGleison Santos, Gleison Santos PPGI/UNIRIO, Federal University of Rio de Janeiro State – Rio de Janeiro, RJ, BrazilSearch for more papers by this author Eliezer Dutra, Corresponding Author Eliezer Dutra eliezer.goncalves@uniriotec.br PPGI/UNIRIO, Federal University of Rio de Janeiro State – Rio de Janeiro, RJ, Brazil CEFET/RJ, Celso Suckow da Fonseca Federal Center for Technological Education – Nova Friburgo, RJ, BrazilSearch for more papers by this authorGleison Santos, Gleison Santos PPGI/UNIRIO, Federal University of Rio de Janeiro State – Rio de Janeiro, RJ, BrazilSearch for more papers by this author First published: 17 February 2021 https://doi.org/10.1049/iet-sen.2020.0048AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Agile methods are associated with values, principles, and practices that influence organisational climate. Organisational climate is the meaning employees attach to the policies, practices, and procedures they experience and the behaviours they observe getting rewarded, supported, and expected. A negative climate can influence team members on aspects such as motivation, ability to make good decisions, and willingness to innovate. Conversely, a positive climate can influence project success and the competitiveness of organisations. The authors investigate how organisations considered assessing the organisational climate of Agile teams, and which are the benefits and the difficulties associated with this assessment. They conducted a qualitative study on five Brazilian organisations. They interviewed key personnel involved with organisational climate assessments. They identified 16 benefits and nine difficulties of organisational climate assessments. Based on the results and the literature, they formulated two propositions representing pitfalls that can hinder organisational climate assessments of Agile teams. Organisations should use instruments adapted to the Agile culture to improve the ability to diagnose the organisational climate and involve the Agile team in the climate management activities to ease the identification of the team's perception about project performance and product quality indicators. 1 Introduction Competitiveness between organisations requires the speedy launch of innovative products and services into the market. Therefore, organisations are challenged to select management models that provide employees with independence for decision-making, cooperation, and team learning, thus transferring the perspective of authority and control to the employee [1]. Agile methods have shown good prospects for the development of self-management, autonomy, continuous learning, and innovation in software development teams [2, 3]. Many organisations define their software development processes based exclusively on Agile methodologies or hybrid strategies, i.e. a combination of traditional and Agile practices. However, 'being agile' does not depend only on the Agile methodologies or practices adopted, but on a set of factors that involve the rapid change in planning and active involvement of the customer [4]. Moreover, many problems are often hidden in the management layers and the bureaucracy of organisations [3]. The rapid change in iterations planning requires a high degree of autonomy, communication, cooperation, and self-organisation from the team members [5, 6]. Moreover, planning must consider the collaboration with the customer, taking into account business value and team capacity [5]. Support, trust, and flexibility are required at all levels of management in the organisation, so the planning is effective. Therefore, the path to agility involves, in addition to adopting practices, changing the team members' behaviour, active customer involvement, and support and trust of all management levels at the organisation. Nevertheless, the adoption and use of Agile values, principles, and practices can be negatively impacted by some issues [7, 8], such as lack of skills/experience with Agile methods, minimal collaboration and knowledge sharing, the existence of hybrid project standards, inconsistent processes and practices across teams, inadequate management support and sponsorship. To understand these issues, one should seek Agile team members' opinions on their working conditions. Organisational climate surveys can help with this matter. The organisational climate assessment allows the capturing of the perception, satisfaction, or preferences of the development team in terms of their working conditions. A negative organisational climate can have adverse effects on software development team members about aspects such as motivation, the ability to make good decisions, and willingness to innovate while also affecting a project's success [9, 10]. Considering Agile teams, organisational climate assessments allow, for instance, to identify ineffectiveness of leadership, measure the autonomy, collaboration, or communication of the team, identify the lack of support from the various levels of management, notice inconsistencies in the use of ceremonies, and capture the teams' preferences among the various existing practices or tools etc. Both industry and academia have shown interest in the organisational climate and its effects on teams and on organisations for decades [11]. This interest encompasses not only business or human resource management perspectives but also a software engineering perspective [12]. Therefore, the organisational climate should be measured; moreover, proper organisational climate management can help organisations to gain a competitive advantage [11]. An effective organisational climate assessment depends on theoretically established items that address specific management demands for a period [13]. Many organisational climate assessments use items proposed in the first half of the 20th century [11, 13], which can impact their use by Agile teams. Therefore, both the assessment process and items should enable critical reflections on the team activities, the behaviours of those involved, the organisation processes, and the culture. There is a gap in the literature regarding the investigation on how organisational climate assessments apply to Agile teams, how the process is, how the instruments are, what the benefits or difficulties are. The majority of the studies have focused on investigating factors that influence the organisational climate of development teams in general [10]. We present a qualitative study aimed at understanding how organisations assess the organisational climate of Agile teams, and which are the benefits and the difficulties associated with this assessment. We interviewed five personnel involved with organisational climate assessments. We used the interview data to define two propositions based on the observed relationships and the interpretative understanding generated, as suggested in [14]. The propositions describe pitfalls that hinder the organisational climate management. The pitfalls can trigger those responsible for managing the team's organisational climate to avoid similar situations by improving the adopted procedures. Human resources (HR) managers, project managers, and Agile leaders must (i) adapt their instrument to the values, principles, practices, and Agile roles; (ii) decrease the time interval and the number of items evaluated; and (iii) include discussions of analysis of results and definition of control actions with the team members. Moreover, practitioners must consider the factors identified in the literature that specifically influence the organisational climate of Agile teams. Furthermore, the established propositions can motivate researchers to identify practical issues to be addressed in future research studies. This paper is organised as follows: Section 2 provides background on organisational climate assessments. Section 3 presents the methodology and study design. Section 4 shows the results. Section 5 presents pitfalls associated with the execution of organisational climate assessments. Section 6 addresses the study limitations and threats to validity. Section 7 shows our final considerations. 2 Background and related work 2.1 Organisational climate There are many definitions of organisational climate [11, 15, 16]. Schneider and Barbera [11] defined organisational climate as the meaning employees attach to the policies, practices, and procedures they experience and the behaviours they observe getting rewarded, supported, and expected. Verbeke et al. [16] identified 32 different definitions of organisational climate in the literature. After that, they identified categories and dimensions (i.e. clusters) commonly associated with the term (number of citations in parentheses): organisation (30), members (26), perceptions (21), characteristics (15), behaviour (15), shared (11), influence (9), and practices (8). After applying statistical inference on the identified clusters (i.e. categories and dimensions), the authors defined organisational climate as a reflection of the way people perceive and come to describe the characteristics of their environment [16]. Lenberg et al. [15] proposed an update of the term behavioural software engineering (BSE) to characterise concepts of human aspects of software engineering. The authors argue that there are knowledge gaps in the research area of BSE and that earlier research has focused on a few concepts (motivation, organisational change, cognitive style, job satisfaction etc.) which have been applied to a limited number of software engineering areas. Among these concepts, organisational climate is defined as the recurring patterns of behaviour, attitudes, and feelings that characterise life in the organisation [15]. The assessment of the organisational climate is part of organisational development. It is commonly associated with diagnostic activities, development or selection of the approach to be used, data analysis, problem-solving action plans, and monitoring activities [17-19]. The diagnostic stage aims to identify indicators and dimensions associated with the organisational climate or issues that interfere with the way the organisation operates. Based on this, an approach to assess the organisational climate is developed. An assessment instrument is defined considering the issues and indicators previously identified. The assessment instrument comprises items (i.e. questions) associated with a measurement scale. These items are usually analysed in dimensions. Once the instrument is validated, data collection can begin. During analysis, workforce groups are formed to understand the collected data and mediate differences among those involved in its interpretation. Finally, actions to deal with the issues identified and stimulate organisational change are defined, executed, and monitored. 2.2 Studies investigating organisational climate in Agile teams Many studies have investigated organisational climates in software development teams [10]. Many of the studies published, however, consider the teams without mentioning the base of the development process and utilise climate assessment models of a general nature. The studies we identified [10, 20-22] investigated factors, which influenced the development of organisational climates in Agile teams. Soomro et al. [10] conducted a systematic review of the literature on climate, personality, and performance of software development teams. The authors report that the development of an Agile team's organisational climate includes factors that represent the involvement of the whole team, Agile values (trust, openness, and respect during team interaction, for example), Agile practices (feedback and negotiations), a culture favouring action and change (in terms of bringing progress and improvement to team activities) and collective thinking. Acuña et al. [20] conducted an empirical study to investigate the relation between innovation climate [23] and product quality. The study was carried out among 35 groups, with three students in each group. The teams utilised a process adapted from the extreme programming (XP) method. The results show the dimensions of vision and participative safety are significantly related to better software. Acuña et al. [20] recommended following an organisation's climate as a quality indicator for the software to be delivered. Serrador et al. [22] identified organisational climate factors that influence a project's success. Three dimensions (top management support, sufficient resources and willingness to adapt) were identified and tested to measure their ability to predict different aspects of project success. The results of a data regression analysis from 449 projects showed that the climate for project success was a significant and strong predictor of both stakeholder success and budget/time success. Top management support and willingness to adapt positively influenced stakeholder success; top management support positively influenced budget/time success. Interestingly, budget flexibility, an element within the sufficient resources dimension, had a negative relation to both stakeholder and budget/time success. Serrador et al. [22] claim that organisations that want to boost the success of their projects must focus on the climate development of (i) senior management support, (ii) engagement of interested parties, (iii) dedicated team, (iv) Agile methods support, (v) frequent meetings with product owners, and (vi) a good team attitude to accept changes. Vishnubhotla et al. [21] investigated the association between personality traits and the factors related to team climate within the context of Agile teams working in a telecom company. The authors observed a statistically significant positive correlation between the personality traits openness to experience and support for innovation. Additionally, agreeableness was found to be positively correlated with the overall team climate. In brief, a person's ability to easily get along with team members (i.e. agreeableness) has a significant positive influence on the perceived level of team climate. The authors suggest there are other human factors, in addition to personality traits that should also be investigated about their relationship with team climate. In summary, Acuña et al. [20] considered students in the experiment, Serrador et al. [22] identified the factors with project managers, and only Vishnubhotla et al. [21] investigated the climate with Agile team professionals. The studies [20, 21] used the Team Climate Inventory instrument [23] through four dimensions (participative safety, support for innovation, team vision, and task orientation) to measure the climate. Therefore, there is a need for further studies to understand the practice how organisations carry out the climate assessment in Agile teams, how the instrument is, and what the measured dimensions are. 2.3 Formation of the organisational climate in Agile teams Agile software development comprises a set of iterative and incremental software engineering methods that are inspired by the 'Agile philosophy', as per the Agile manifesto [2, 3, 5, 24], and operationalised through Agile practices [3, 5]. The Agile manifesto provides values and principles that represent behaviours that are demanded from the team an ability for a quick assessment, feedback, and change [2]. The way in which top management, coordinators, leaders, and the team utilise and believe in these values and principles influences the software development Agile team's organisational climate [5, 10, 22, 25]. Within this context, the difference in terms of dynamics in traditional and Agile projects makes it necessary for the assessment of the organisational climate to be adapted to an Agile context. Some differences are discussed below. With regard to management style, in traditional development, command and control are associated with the process, the manager, the coordinator, and top management. In contrast, Agile development focuses on leadership and collaboration [26]. Control and command relating to the development process and behaviour are transferred to the Agile team. As for management style, Schneider and Barbera [11] claim that leadership is central to climate development and maintenance. Thus, climate management must take into account the role of a servant leader and the individual leadership of each member. The leader's role in an Agile team is performed by an Agile facilitator, a servant leader [5]. This role can be referred to as that of a facilitator, a coach, and a team coordinator [5, 25]. The Agile facilitator helps the team to achieve high performance. In Scrum, for instance, among other attributions, the Scrum Master promotes the team's adherence to Scrum practices (e.g. events, artefacts, roles, and rules) [25]. Regarding the customer activities, in traditional methods, customer activities are often limited to supply and clarify requirements. In Agile development, in addition to providing, clarifying, and prioritising requirements, it is part of the customer's responsibility to plan the iteration and negotiating deliveries with the team [5]. In Scrum, for example, this role is played by the Product Owner. Another key difference relates to the desired organisational structure. While in traditional development the organisational structure is mechanistic, i.e. bureaucratic with high formalisation [26], in Agile development it is organic, i.e. flexible and participative encouraging cooperative social action [26]. Wlodarkiewicz-Klimek [27] claims that companies that want to improve business agility must ensure there is a climate to make the organisational structure of management more flexible, adaptable to emerging changes. To accomplish this, the team reflects, at regular intervals, on how to become more effective, then tunes and adjusts its behaviour accordingly [2]. Melo et al. [28], in the study about motivation in the Agile team, showed that lack of bureaucracy during the development process motivates Agile teams. The unmotivated Agile team member influences negatively the formation of the organisational climate in various dimensions. For instance, retention, project delivery time, productivity, budgets, and project success may be affected. The literature identified [5, 10, 11, 22, 25, 28] makes it possible to understand that the organisational climate in Agile teams comprises (i) whole team involvement; (ii) Agile values and practices; (iii) culture of action and change; (iv) collective thinking; (v) servant leadership; (vi) Agile method support; (vii) frequent meetings with product owners; (viii) a positive team attitude to accept changes; (ix) active customer involvement; (x) senior management support; (xi) engagement of the interested parties, and (xii) dedicated team. The 12 main elements identified in the literature represent a grouping of critical factors that influence the formation of the organisational climate of Agile teams. In light of the significance of these factors, it is important to hear the voice of practitioners responsible for managing the organisational climate to understand how organisations assess the organisational climate of Agile teams. 3 Methodology We used an approach qualitatively to collect and analyse the data [14, 29]. Qualitative methods are designed to assist researchers in understanding phenomena in context [14, 29]. Qualitative researchers typically gather a variety of data of different sorts from interviews to documents to observations and so forth [14, 29]. Thereby, we considered using document analyses and interviews. Document analysis includes studying official publications, reports, or other documents [29]. Patton [29] states that interviews allow us to capture the opinion of participants, staff, and other personnel involved in specific programmes or processes of the organisation. We used the interviews for descriptive and explanatory purposes. Descriptive interviews are used to provide a rich description of a phenomenon as perceived by individuals [14, 29]. Exploratory interviews are typically used to define questions, propose new theory constructs, and/or build new theories. Usually, propositions or hypotheses are generated based on the observed relationships or the interpretive understanding generated [14]. 3.1 Study design The objective of the study is to investigate how organisational climate assessments are carried out and deal with Agile team contexts. Moreover, we want to identify pitfalls associated with these assessments and propose new research studies. To accomplish our goal, we interviewed key personnel involved with organisational climate assessments. To accomplish the study goal, we defined the following research questions (RQ): RQ1: How do organisational climate assessments allow for the identification of behaviour corresponding with Agile methodologies and the Agile culture? RQ2: Which benefits and difficulties are associated with the performance of organisational climate assessments? RQ3: How do organisational climate assessments identify potential influences of software quality and Agile team performance? To answer the RQs proposed, we formulated the questions (Q) given in Table 1. Table 1. RQs and questions considered in the interviews Research question Questions considered warm-up Q1. How does your organisation assess the organisational climate of agile software development teams? How is the process? What is the frequency? How is the instrument? What are the dimensions? RQ1 Q2. Does the instrument cite any human factor? Which ones? Q3. Does the instrument consider the agile culture? How? Q4. Does the assessment instrument capture the perception of good and bad behaviours associated with the agile methodologies? RQ2 Q5. Which benefits do organisational climate assessments provide to your organisation? Q6. Which difficulties are faced to assess the organisational climate of agile software development teams in your organisation? RQ3 verbatim We used warm-up questions to determine the organisations' context, the organisational climate assessment process, and the instrument used. The objective of RQ1 is to understand how organisational climate assessments represent behaviours (evaluation items) compatible with an Agile culture. The term 'behaviour' in RQ1 was used because it is commonly used in definitions of organisational climate [16]. The term 'human factor' in Q1 was used because it represents the physical or cognitive features, or social behaviour, of a person [30]. For that reason, the expression 'human factor' was used to facilitate the identification of behaviours, feelings, or attitudes assessed in the organisational climate survey. The use of the terms 'behaviour' or 'human factor' in the RQs does not exclude, for example, items or dimensions that evaluate processes or practices. Organisational climate assessments focus on employees' perceptions of different elements. Thereby, evaluation items are often described through behaviours, attitudes, feelings, or preferences. For example, Açıkgöz and Günsel [31] used the item 'The norms and procedures in my work team … Help our team to function better' to assess the team's perception of the informal structure of the organisational process. The objective of RQ2 is to find the benefits (Q5) and difficulties (Q6) associated with carrying out an organisational climate assessment. Through the answers given to Q5, proposals were established, which represent pitfalls and new research studies. To answer RQ3, we asked participants the research question verbatim. The objective of this question is to find how climate assessment results enable the identification of possible influences on the quality indicators of the software developed or the Agile team performance. Product quality indicators and team performance are measured based on the analysis of the artefacts produced and the effort made in projects. It is important to notice that organisational climate assessments aim to capture the opinion or perception of employees about his/her own work, colleagues, or the organisation itself. Thus, even if the team's estimates are considered correct as measured by project indicators, climate assessment could capture intentional deviations through developers' perception. For instance, a team member may think the productivity of his/her colleague is affected by long pauses or the use of working hours to attend to personal issues. 3.2 Data collection and analysis We sent email invitations to Brazilian organisations that execute organisational climate assessments frequently and have software development teams using Agile methodologies. The goals of the study and the definition of organisational climate were present in the invitation letter and reinforced verbally at the interview. All participants were involved in organisational climate management activities. Therefore, the participants had no doubts about the questions. In addition to the invitation letter, we also included a consent form to comply with ethical principles. The consent form showed the goals of the study, the participants' rights, and the privacy issues. The participants work on organisations that perform organisational climate assessments frequently and have software development teams using Agile methodologies. Each interview lasted an average of 30 min. For reasons of privacy, only two companies made the instrument and the last executive reports available in the assessment for us to analyse. After the warm-up questions, we asked questions based on the RQs (see Table 1). We asked the interviewees to talk freely about the inquired topics. The thematic analysis [32, 33] was used to answer question RQ2. Fig. 1 shows the thematic analysis process we used to code the data. Fig. 1Open in figure viewerPowerPoint Thematic analysis process used (adapted from [32, 33]) The steps shown in Fig. 1 were used for identifying, analysing, and reporting patterns within data. Thus, we identified codes and categories relevant to the answers to the question RQ2. The first author performed the thematic analysis by examining interview transcripts and/or the raw audio recording. We used ATLAS.ti (https://atlasti.com/) as a supporting tool. The first author defined the codes associated with the text/audio (quotes). Later, he defined categories to group similar codes. All codes and categories were discussed and revised by the second author. To verify the coherence of (i) the codes with the transcripts and (ii) the categories with the codes, the second author used the following questions [32] 'Are there clear, evident connections between the text and the codes?' and 'Are categories internally coherent, consistent, and distinctive?' Necessary adjustments to the description of codes and categories were made by the first author. Examples of the coding process can be found in Section 4. 4 Study results Next, we present findings related to the warm-up and RQs. We provide examples of quotes from participants who supported our findings. 4.1 Characterisation of the participating organisations Understanding the context in which the data is collected is crucial to interpret data collected in any empirical study [14]. Some of the information describing the organisations was obtained through warm-up questions (Table 1). Companies' names are omitted for confidentiality. We considered both small and large organisations. Table 2 gives an overview of the participating organisations and interviewees. In Table 2, the numbers in the 'Employees' column represent the total number of organisation employees and (in parentheses) the number of employees in the IT department. The column 'Climate dimension' represents the grouping of items to measure the dimension described (see the last paragraph in Section 2.2). The 'Climate dimension' was extracted through the executive reports of Organisations C and E. For Organisations A, B, and D, the dimensions were extracted using the analysis of open-ended questions. Table 2. Participants and organisations involved Participant Role Organisation type/software focus Employees (IT specific) Software processes used Climate survey frequency Climate dimension P1 agile team coordinator (A) startup; banking +20 (4) Scrum undeter­mined fre­quency leadership; culture; owner sentiment; initiative; career progress opportunities P2 agile team coordinator (B) insurance +10,000 (N/A) Waterfall; RUP; Agile yearly engagement (identification); leadership; team recognition; career progress opportunities; pay; clarity of objectives P3 agile team coordinator (C) E-commerce +10,000 (+1000) Scrum; Kanban weekly engagement (identification); wellbeing; career; professional development; feedback; recognition; leadership; diversity P4 HR manager (D) software house; accounting; real estate +1500 (+650) Scrum yearly satisfaction; process; benefits; leadership; pay P5 agile