# Large-Scale Agile Transformations for Software Quality Assurance: An Empirical Case Study from Pakistan

Authors: Kamran Wadood; Natasha Nigar; Muhammad Kashif Shahzad; Shahid Islam; Abdul Jaleel; Douhadji Abalo
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Software quality plays an important role in the easy and quick adoption of a software product by the end users. Agile methods have proven to play an effective role in ensuring software quality; however, the public sector is hesitant in its adoption. This study evaluates the adoption of agile methods in the public sector in-house software development and capitalization of potential benefits. A quantitative approach is used with 28 survey questionnaires based on budget, time, quick debugging and correction, maintenance, easy testing, and installation as software quality components to assess the employability of agile methods. The questionnaire was served to 216 information technology (IT) professionals (project managers: 6%, developers: 61%, quality assurance (QA) leads: 8%, and testers: 25%) from the public sector companies having experience in software development using agile and waterfall methods. The quality components and hypotheses are evaluated using theT-test and chi-square test, respectively, with a 95% confidence interval. The results highlight the benefits of using agile methodologies for software quality components in the public sector. Additionally, the findings demonstrate how agile software development approaches significantly affect the quality of software products and successful delivery within budget and deadline.


---

## Body

## 1. Introduction

In the modern software industry, the prime objective is the delivery of high-quality software in a shorter time. The software project’s success and the satisfaction of customers’ expectations are widely measured by the quality of the software [1]. A non-systematic software development approach applied to a large-scale software project will result in software products that have high costs with low quality. Therefore, an approach towards software development is a key contributor in the decision related to the quality of the software [2]. IT initiatives, about 5 to 10% of the organizations’ total revenues, have made this even more important [3].As per De Feo [4], software with high-quality has potential benefits for both the customer and the organization because the organization becomes more responsive, gains an edge over its competitors, and reduces its cost for development and marking time. Delivering low-quality software limits the company’s growth. Consequently, organizations could risk their reputation, and survival will be at stake in dynamic business settings.Traditional software methodologies are plan-driven, starting with the requirements, elicitation, and documentation, which leads to the architectural and high-level design development and inspection. The waterfall, V-model, and rational unified process (RUP) are examples of processes that follow a series of steps, including defining the requirements, building the solution, testing, and deployment. These methods cannot deal with changing customer requirements, raising concerns about quality problems [5].Since the last decade, agile methods have become a popular trend for companies to improve their performance, focusing on software quality. Many companies’ transformation from traditional software development to agile software development has dealt with complex projects with ill-defined requirements, high customer satisfaction, low defect rates, and fast development time with evolving customer needs. In these methods, software quality is ensured by customer collaboration. Now, the evolution of the end software product or service is actively shaped and guided by the customers/stakeholders; rather, they prefer to stay at the fringes of software development. However, its benefits are still unknown for successful project delivery in large public software development organizations while confirming quality. This is because of nasty attitudes, hierarchical, bureaucratic management styles, and willingness for change in these organizations [6].This work has assessed the adoption of agile methods in public sector software development to improve the quality of the software and successful delivery. The key objective is to present a sustainable e-Governance model to ensure service delivery to the citizens. We selected nine public-sector companies with in-house software development to study the effect of quality factors, including “budget,” “time,” “quick debugging and correctness,” “maintenance,” “easy testing,” and “installation.” The results show that agile software development methods significantly impact the software product quality and successful delivery within budget and time. The key contributions of this study are as follows:(i)
Employability of the Agile method: we employ and access the agile method in a large-scale public sector environment to ensure quality(ii)
Result generation: a survey was performed, based on 28 questions and focusing on software quality components, by IT professionals from nine companies (involved in software development, under the umbrella of the Water and Power Development Authority (WAPDA), which is one of the largest public organizations in Pakistan)This paper is structured into six sections. Section2 reviews the background and related work. Section 3 proposes a conceptual framework following the results and discussions in Section 4. Section 5 presents the threats to validity. Finally, Section 6 concludes with future directions.


## 2. Literature Review

### 2.1. Background

The section is categorized into software quality, process models, and traditional and agile development methods. The key focus is to find the factors that influence software quality and develop a conceptual framework for it.


### 2.2. Software Quality

It is observed that a software product that is delivered within time and budget while performing its target functions correctly and efficiently still lacks quality. Consequently, the software product is hard to understand, difficult to use and maintain, easy to misuse, machine-dependent, and difficult to integrate with other software. The software quality is defined in the literature from different views, which are stated as follows:(i)
Customer view: it is characterized as an extent to which the product, process, or service fulfils the requirements [7–9](ii)
Product view: in this context, the quality measures the unvalued features contained in every valued feature [9](iii)
Engineering view: the quality is characterized as the extent to which a particular item affirms an architecture or requirement [9](iv)
Value view: the level of tended fulfillment of customer expectations at an affordable cost under variation [9]Software quality assurance is more customer-centric and can be attributed to a software product that is free of defects, delivered on time within budgetary constraints, fulfills requirements and/or desires, and can be maintained [2].


### 2.3. Quality Models

In this section, the popular quality models are presented. These are employed along with quality standards to ensure compliance with high-quality software requirements. Jim McCall introduced a quality model that fundamentally revolves around system and process developers. The emphasis is on efforts to bridge the difference between customers and software developers by ensuring quality aspects or features and paying considerable attention to the requirements [10]. Another model is Boehm’s model, which complements McCall’s quality model. In these models (Figures 1(a), 1(b), and 2), the subjective methodology is used to characterize (with focus on three levels) the primitive attributes, which complete the quality definition for high-quality software products. In addition to these, International Organization for Standardization (ISO) 9126 is the quality standard that projects quality features to be used to assess six important areas, as mentioned in Figure 3.Figure 1
Software quality models: (a) McCall quality model and (b) ISO9126 quality model.
(a)(b)Figure 2
Software quality model: Boehm quality model.Figure 3
Quality attributes of ISO 9126.


### 2.4. Traditional and Agile Software Development Approaches

#### 2.4.1. Waterfall Model

The waterfall is one of the oldest software development models proposed by Royce in 1970 [11]. According to this model, the development phases are sequential, where a new phase starts upon completion of the current phase. The project manager expects tasks to be completed as soon as possible once they are started; however, it has been observed that project information and developers’ knowledge improve with time. The inherent compliance to this model requires phases to be started even with missing or incomplete information. This model best fits the software projects where user requirements are properly documented and locked. It should be avoided for large-scale enterprise software projects where requirements regularly evolve. Moreover, this model does not incorporate a quality assurance loop between completed phases.


#### 2.4.2. Agile Methods

The “agile” concept refers to the methods and practices based on values and principles presented in the Agile Manifesto 2001 [12]. The software development approaches derived from “agile” have the ability to adapt under uncertainty. They have replaced the traditional “waterfall” approach based on being time-boxed and iterative, where software is developed in increments called sprints, compared to delivering it as a complete package.The key elements of the “agile” manifesto are presented as follows:(i)
user engagement over complex processes(ii)
functional software product against detailed documented product(iii)
user engagement in contractual negotiation(iv)
quick adoption of changeThe “agile” approach emphasizes people and communication rather than processes and tools. The focus is on functional software that meets the expectations of the customers rather than comprehensive documentation. In the “agile” approach, tasks are broken into smaller activities to be completed incrementally with daily meetings to ensure that the project is on track. There exist many methodologies under “agile,” e.g., scrum [13], extreme programming (XP) [14], feature-driven development [15], and many more. The scrum and XP methodologies are widely used. In the light of one survey, 66.7% of software houses in Pakistan use the scrum software process model as shown in Figure 4.Figure 4
Software models usage in Pakistan software industry.The software houses are adopting agile software development with feasible inclusion of changes, even in the last phases of the project [16]. This offers a competitive edge among all the software industry competitors. This highlights that the public sector in-house software development has yet to benefit from the “agile” methodologies. They currently rely on traditional software development approaches like “waterfall” to support the sustainable e-Governance model to serve the citizens under the same model.


### 2.5. Related Work

Nerurkar and Das [17] discussed and analysed the need for an agile project management framework for large scale projects in government and public sectors. However, they did not discuss any quality attribute. Bolhuis [18] evaluated that how large-scale agile can be effectively adopted and scaled up in Dutch public sector organizations. They concluded that their results might not be applicable for all Dutch public sector organizations. Mohagheghi and Lassenius [19] presented an organizational approach in adopting the agile in a large public organization. The study in [20] presented the challenges collected from the past studies of IT project implementation in the government sector considering the agile method. They identified 20 challenges and categorized into technology, organization, environment, and individual context. The work in [21] addressed the agile transformation in large companies with existing software product lines and proposed a transformation model. Fangmann et al. [22] also identified the challenges posed by the adoption of agile practices in public administration and how these challenges can be overcome. Wadood et al. [23] analysed the software quality components, e.g., budget and time for agile development in a public sector environment. Bousdekis and Kardaras [24] identified the challenges of adopting digital technologies (particularly agile) in the public sector and in local governments. In [25], authors focused on agile developments for mission critical systems in the public sector. Vacari and Prikladnicki [26] presented a systematic literature review for adopting agile methods in the public sector. They concluded that agile methods could be adopted in the public sector. However, not all the implications of adopting agile methods in the public sector are widely known.The above literature highlights the fact that most of the studies have identified the challenges in adopting agile methods in the public sector without consideration of quality characteristic. It is imperative to identify the quality attribute implications in agile methods for large public sector organization; therefore, it motivates our research to consider this factor for adopting agile in large public sector organization ensuring software quality.


## 2.1. Background

The section is categorized into software quality, process models, and traditional and agile development methods. The key focus is to find the factors that influence software quality and develop a conceptual framework for it.


## 2.2. Software Quality

It is observed that a software product that is delivered within time and budget while performing its target functions correctly and efficiently still lacks quality. Consequently, the software product is hard to understand, difficult to use and maintain, easy to misuse, machine-dependent, and difficult to integrate with other software. The software quality is defined in the literature from different views, which are stated as follows:(i)
Customer view: it is characterized as an extent to which the product, process, or service fulfils the requirements [7–9](ii)
Product view: in this context, the quality measures the unvalued features contained in every valued feature [9](iii)
Engineering view: the quality is characterized as the extent to which a particular item affirms an architecture or requirement [9](iv)
Value view: the level of tended fulfillment of customer expectations at an affordable cost under variation [9]Software quality assurance is more customer-centric and can be attributed to a software product that is free of defects, delivered on time within budgetary constraints, fulfills requirements and/or desires, and can be maintained [2].


## 2.3. Quality Models

In this section, the popular quality models are presented. These are employed along with quality standards to ensure compliance with high-quality software requirements. Jim McCall introduced a quality model that fundamentally revolves around system and process developers. The emphasis is on efforts to bridge the difference between customers and software developers by ensuring quality aspects or features and paying considerable attention to the requirements [10]. Another model is Boehm’s model, which complements McCall’s quality model. In these models (Figures 1(a), 1(b), and 2), the subjective methodology is used to characterize (with focus on three levels) the primitive attributes, which complete the quality definition for high-quality software products. In addition to these, International Organization for Standardization (ISO) 9126 is the quality standard that projects quality features to be used to assess six important areas, as mentioned in Figure 3.Figure 1
Software quality models: (a) McCall quality model and (b) ISO9126 quality model.
(a)(b)Figure 2
Software quality model: Boehm quality model.Figure 3
Quality attributes of ISO 9126.


## 2.4. Traditional and Agile Software Development Approaches

### 2.4.1. Waterfall Model

The waterfall is one of the oldest software development models proposed by Royce in 1970 [11]. According to this model, the development phases are sequential, where a new phase starts upon completion of the current phase. The project manager expects tasks to be completed as soon as possible once they are started; however, it has been observed that project information and developers’ knowledge improve with time. The inherent compliance to this model requires phases to be started even with missing or incomplete information. This model best fits the software projects where user requirements are properly documented and locked. It should be avoided for large-scale enterprise software projects where requirements regularly evolve. Moreover, this model does not incorporate a quality assurance loop between completed phases.


### 2.4.2. Agile Methods

The “agile” concept refers to the methods and practices based on values and principles presented in the Agile Manifesto 2001 [12]. The software development approaches derived from “agile” have the ability to adapt under uncertainty. They have replaced the traditional “waterfall” approach based on being time-boxed and iterative, where software is developed in increments called sprints, compared to delivering it as a complete package.The key elements of the “agile” manifesto are presented as follows:(i)
user engagement over complex processes(ii)
functional software product against detailed documented product(iii)
user engagement in contractual negotiation(iv)
quick adoption of changeThe “agile” approach emphasizes people and communication rather than processes and tools. The focus is on functional software that meets the expectations of the customers rather than comprehensive documentation. In the “agile” approach, tasks are broken into smaller activities to be completed incrementally with daily meetings to ensure that the project is on track. There exist many methodologies under “agile,” e.g., scrum [13], extreme programming (XP) [14], feature-driven development [15], and many more. The scrum and XP methodologies are widely used. In the light of one survey, 66.7% of software houses in Pakistan use the scrum software process model as shown in Figure 4.Figure 4
Software models usage in Pakistan software industry.The software houses are adopting agile software development with feasible inclusion of changes, even in the last phases of the project [16]. This offers a competitive edge among all the software industry competitors. This highlights that the public sector in-house software development has yet to benefit from the “agile” methodologies. They currently rely on traditional software development approaches like “waterfall” to support the sustainable e-Governance model to serve the citizens under the same model.


## 2.4.1. Waterfall Model

The waterfall is one of the oldest software development models proposed by Royce in 1970 [11]. According to this model, the development phases are sequential, where a new phase starts upon completion of the current phase. The project manager expects tasks to be completed as soon as possible once they are started; however, it has been observed that project information and developers’ knowledge improve with time. The inherent compliance to this model requires phases to be started even with missing or incomplete information. This model best fits the software projects where user requirements are properly documented and locked. It should be avoided for large-scale enterprise software projects where requirements regularly evolve. Moreover, this model does not incorporate a quality assurance loop between completed phases.


## 2.4.2. Agile Methods

The “agile” concept refers to the methods and practices based on values and principles presented in the Agile Manifesto 2001 [12]. The software development approaches derived from “agile” have the ability to adapt under uncertainty. They have replaced the traditional “waterfall” approach based on being time-boxed and iterative, where software is developed in increments called sprints, compared to delivering it as a complete package.The key elements of the “agile” manifesto are presented as follows:(i)
user engagement over complex processes(ii)
functional software product against detailed documented product(iii)
user engagement in contractual negotiation(iv)
quick adoption of changeThe “agile” approach emphasizes people and communication rather than processes and tools. The focus is on functional software that meets the expectations of the customers rather than comprehensive documentation. In the “agile” approach, tasks are broken into smaller activities to be completed incrementally with daily meetings to ensure that the project is on track. There exist many methodologies under “agile,” e.g., scrum [13], extreme programming (XP) [14], feature-driven development [15], and many more. The scrum and XP methodologies are widely used. In the light of one survey, 66.7% of software houses in Pakistan use the scrum software process model as shown in Figure 4.Figure 4
Software models usage in Pakistan software industry.The software houses are adopting agile software development with feasible inclusion of changes, even in the last phases of the project [16]. This offers a competitive edge among all the software industry competitors. This highlights that the public sector in-house software development has yet to benefit from the “agile” methodologies. They currently rely on traditional software development approaches like “waterfall” to support the sustainable e-Governance model to serve the citizens under the same model.


## 2.5. Related Work

Nerurkar and Das [17] discussed and analysed the need for an agile project management framework for large scale projects in government and public sectors. However, they did not discuss any quality attribute. Bolhuis [18] evaluated that how large-scale agile can be effectively adopted and scaled up in Dutch public sector organizations. They concluded that their results might not be applicable for all Dutch public sector organizations. Mohagheghi and Lassenius [19] presented an organizational approach in adopting the agile in a large public organization. The study in [20] presented the challenges collected from the past studies of IT project implementation in the government sector considering the agile method. They identified 20 challenges and categorized into technology, organization, environment, and individual context. The work in [21] addressed the agile transformation in large companies with existing software product lines and proposed a transformation model. Fangmann et al. [22] also identified the challenges posed by the adoption of agile practices in public administration and how these challenges can be overcome. Wadood et al. [23] analysed the software quality components, e.g., budget and time for agile development in a public sector environment. Bousdekis and Kardaras [24] identified the challenges of adopting digital technologies (particularly agile) in the public sector and in local governments. In [25], authors focused on agile developments for mission critical systems in the public sector. Vacari and Prikladnicki [26] presented a systematic literature review for adopting agile methods in the public sector. They concluded that agile methods could be adopted in the public sector. However, not all the implications of adopting agile methods in the public sector are widely known.The above literature highlights the fact that most of the studies have identified the challenges in adopting agile methods in the public sector without consideration of quality characteristic. It is imperative to identify the quality attribute implications in agile methods for large public sector organization; therefore, it motivates our research to consider this factor for adopting agile in large public sector organization ensuring software quality.


## 3. Research Methodology

### 3.1. Proposed Conceptual Framework

The quality of a software project is assessed as defects-free, completed within budget and time, and meeting customer requirements [2]. Moreover, the software quality characteristics in public sector in-house software development are quick debugging, correctness, easy testing, and swift single click installation. These factors are evaluated on the software quality for agile approaches in public sector in-house development. A conceptual framework is proposed based on these characteristics in Figure 5.Figure 5
Conceptual framework.


### 3.2. Population and Sample Size

The population in this case study consists of software project managers, software developers, QA testers, and team leads from nine distribution company (DISCO) computer centers. The target designations are chosen because they can provide necessary data and information during a survey for this research. A survey is sent to 216 IT professionals out of 850 employees (Table1).Table 1
Population data.
CompanyNo of employeesTotalSoftware project managersSoftware developersQA leadsSoftware testersPower information technology company101401650216It is pertinent to note that respondents have completed at least one software project using agile and waterfall software process models. The results show that 171 respondents have qualified the set initial criteria (Table2).Table 2
Sample data.
CompanyNo of employeesTotalSoftware project managersSoftware developersQA leadsSoftware testersPower information technology company101031444171


### 3.3. Summary of Questionnaire Design

The researcher gathered data through meetings and by giving questionnaires to QA leads, developers, and testers. The questionnaires helped the researcher acquire data or information from a substantial number of individuals. The questionnaire concentrated on catching information or data in respect of quality factors such as correctness, testability, changeability, and installability (Table3).Table 3
Design of the questionnaire.
AttributeQuality factorsQuestionsAgreement with specificationsCorrectness05–09Consistency in functionalityDefectsPercentage of testTestability10–12 and 15Ratio of effectiveness of testEasiness or simplicityConsistency of codeModification abilityChangeability13–14 and 16–17Errors subsequent to modificationsModification effortsLevel of interconnection and linkAfter installation level of stabilityInstallability18–20


### 3.4. Analysis Method

Intending to improve the software project quality, we formulate the hypotheses in Table4 to access the applicability of the agile software development methodology in large public sector organizations. We follow the below steps to analyze data to support or reject the formulated hypotheses for software developed using agile and waterfall methods.(i)
Organized the data collected through a questionnaire using the statistical package for the social sciences (SPSS) tool, i.e., software developers, QA leads, and software testers(ii)
Built the frequencies against the questions(iii)
To test the derived hypothesis on correctness, testability, changeability, and installability, we used mean andT-test (1-sample) against each quality element(iv)
Used the chi-square test for hypothesis verification on time and budget objectivesTable 4
List of hypotheses.
NoTypeDescription1H0There does not exist any difference in faster debugging and correctnessμAC−μWC=0H1There does exist a difference in faster debugging and correctnessμAC−μWC≠02H0There does not exist a difference in easy testabilityμAT−μWT=0H1There does exist a difference in easy testabilityμAT−μWT≠03H0There does not exist a difference in changeabilityμACh−μWCh=0H1There does exist a difference in changeabilityμACh−μWCh≠04H0There does not exist a difference in installabilityμAI−μWI≠0H1There does exist a difference in easy installabilityμAI−μWI≠0H1There does exist a difference in time to deliver or completeμAD−μWD≠06H0There does not exist a difference in the estimated budget and actual budgetμAB−μWB=0H1There does exist a difference in the estimated budget and actual budgetμAB−μWB≠07H0There does not exist a difference in software projects developed using agile and waterfall methodsμAQ−μWQ=0H1There does exist a difference in software developed with agile and waterfall methodsμAQ−μWQ≠0A full summary of the research methodology has been depicted in Figure6.Figure 6
Research methodology summary.


## 3.1. Proposed Conceptual Framework

The quality of a software project is assessed as defects-free, completed within budget and time, and meeting customer requirements [2]. Moreover, the software quality characteristics in public sector in-house software development are quick debugging, correctness, easy testing, and swift single click installation. These factors are evaluated on the software quality for agile approaches in public sector in-house development. A conceptual framework is proposed based on these characteristics in Figure 5.Figure 5
Conceptual framework.


## 3.2. Population and Sample Size

The population in this case study consists of software project managers, software developers, QA testers, and team leads from nine distribution company (DISCO) computer centers. The target designations are chosen because they can provide necessary data and information during a survey for this research. A survey is sent to 216 IT professionals out of 850 employees (Table1).Table 1
Population data.
CompanyNo of employeesTotalSoftware project managersSoftware developersQA leadsSoftware testersPower information technology company101401650216It is pertinent to note that respondents have completed at least one software project using agile and waterfall software process models. The results show that 171 respondents have qualified the set initial criteria (Table2).Table 2
Sample data.
CompanyNo of employeesTotalSoftware project managersSoftware developersQA leadsSoftware testersPower information technology company101031444171


## 3.3. Summary of Questionnaire Design

The researcher gathered data through meetings and by giving questionnaires to QA leads, developers, and testers. The questionnaires helped the researcher acquire data or information from a substantial number of individuals. The questionnaire concentrated on catching information or data in respect of quality factors such as correctness, testability, changeability, and installability (Table3).Table 3
Design of the questionnaire.
AttributeQuality factorsQuestionsAgreement with specificationsCorrectness05–09Consistency in functionalityDefectsPercentage of testTestability10–12 and 15Ratio of effectiveness of testEasiness or simplicityConsistency of codeModification abilityChangeability13–14 and 16–17Errors subsequent to modificationsModification effortsLevel of interconnection and linkAfter installation level of stabilityInstallability18–20


## 3.4. Analysis Method

Intending to improve the software project quality, we formulate the hypotheses in Table4 to access the applicability of the agile software development methodology in large public sector organizations. We follow the below steps to analyze data to support or reject the formulated hypotheses for software developed using agile and waterfall methods.(i)
Organized the data collected through a questionnaire using the statistical package for the social sciences (SPSS) tool, i.e., software developers, QA leads, and software testers(ii)
Built the frequencies against the questions(iii)
To test the derived hypothesis on correctness, testability, changeability, and installability, we used mean andT-test (1-sample) against each quality element(iv)
Used the chi-square test for hypothesis verification on time and budget objectivesTable 4
List of hypotheses.
NoTypeDescription1H0There does not exist any difference in faster debugging and correctnessμAC−μWC=0H1There does exist a difference in faster debugging and correctnessμAC−μWC≠02H0There does not exist a difference in easy testabilityμAT−μWT=0H1There does exist a difference in easy testabilityμAT−μWT≠03H0There does not exist a difference in changeabilityμACh−μWCh=0H1There does exist a difference in changeabilityμACh−μWCh≠04H0There does not exist a difference in installabilityμAI−μWI≠0H1There does exist a difference in easy installabilityμAI−μWI≠0H1There does exist a difference in time to deliver or completeμAD−μWD≠06H0There does not exist a difference in the estimated budget and actual budgetμAB−μWB=0H1There does exist a difference in the estimated budget and actual budgetμAB−μWB≠07H0There does not exist a difference in software projects developed using agile and waterfall methodsμAQ−μWQ=0H1There does exist a difference in software developed with agile and waterfall methodsμAQ−μWQ≠0A full summary of the research methodology has been depicted in Figure6.Figure 6
Research methodology summary.


## 4. Results and Discussion

The acronyms S, R, and A refer to sample, received, and accepted questionnaires. To collect data from software developers, QA leads, and software testers, 161 questionnaires were circulated. As a result, 139 responses (95%) were received, which are shown in Table5.Table 5
Survey response statistics.
CompanySoftware developersQA leadsSoftware testersTotalSRASRASRAPITC10310095141212443532Total S1031444161Total R1001235147Total A951232139R% age91.30%A% age (with reference to S)86.34%A% age (with reference to R)94.56%The results in Table5 show that the researcher received 147 questionnaires out of 161 questionnaires that were distributed. The response rate is about 91%. Eight respondents did not fulfill the defined criteria because eight questionnaires were ignored in the analysis. Therefore, 139 questionnaires were considered for analysis for this research, and their percentage is about 95%.In Figure7, we represent the diversification of 139 respondents based on their designation. It shows that 68.35% respondents are developers, 8.63% are quality assurance leads, and 23.02% are testers as per valid questionnaires.Figure 7
Respondent statistics as per valid questionnaires.


### 4.1. Assessment of Quality Elements

#### 4.1.1. Quick Debugging and Correctness

To access this quality characteristic, questions’ responses (5–9) are evaluated using a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-1. The results in Table 6 show that significant values are less than 0.05, so the null hypothesis is not accepted. It indicates that there does exist a difference in faster debugging and correction for the software developed using agile and waterfall. The visual representation is also presented in Figures 8 and 9 from the developers’ point of view.Table 6
T-test statistics for questions (5–9).
Statement/QuestiontdfSig. 2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersUser_Expectation29.6319403.6113.373.85Requirement_Capture34.2319403.7373.523.95System_Design32.6089403.7473.523.98System_Implementation32.2599403.6423.423.87Faults_Free38.6629403.7893.593.98QA LeadsUser_Expectation13.0531104.0833.394.77Requirement_Capture17.111103.9173.414.42System_Design15.6541103.5833.084.09System_Implementation17.8381104.0833.584.59Faults_Free14.1991104.0833.454.72TestersUser_Expectation20.9433103.8133.444.18Requirement_Capture15.7743103.4693.023.92System_Design20.5863103.7813.414.16System_Implementation17.5173103.6563.234.08Faults_Free19.3953103.7193.334.11Figure 8
Histograms for the correctness quality factor from developers’ point of view: (a) user expectation, (b) requirement capture, (c) system design, and (d) system requirement.
(a)(b)(c)(d)Figure 9
Histograms for the correctness quality factor from developers’ “fault free” point of view.


#### 4.1.2. Quick Testability

To access this quality characteristic, questions’ responses (10–12,15) are also assessed on a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-2. The results in Table 7 indicate that the null hypothesis is not accepted as significant values are less than 0.05. Therefore, it is concluded that a difference exists in easy testability quality characteristics for software developed with waterfall and agile. The visual representation of these results is also shown in Figure 10.Table 7
T-test statistics for questions (10–12, 15).
Statement/questiontdfSig.2 tailedMean diff95% confidence interval of differenceLowerUpperDevelopersExecution_Test_Script29.6759403.4633.233.69Coding_Standards29.089403.5053.273.74No_Complex_Structure33.0539403.7583.533.98Low_Interaction35.9779403.7583.553.97QA LeadsExecution_Test_Script11.1531103.5002.814.19Coding_Standards17.1101103.9173.414.42No_Complex_Structure10.3831103.5002.764.24Low_Interaction9.7501103.6672.844.49TestersExecution_Test_Script16.5663103.5633.124.00Coding_Standards20.2563103.7503.374.13No_Complex_Structure17.7233103.6253.214.04Low_Interaction16.0913103.6563.194.12Figure 10
Histograms for the testability quality factor from developers’ point of view.
(a)(b)(c)(d)


#### 4.1.3. Changeability

To access this quality characteristic, questions’ responses (13–14, 16–17) are evaluated using a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-3. The results in Table 8 indicate that the null hypothesis is not accepted as significant values are less than 0.05. Therefore, it is concluded that there exists a difference in changeability for the software developed with waterfall and agile. The visual representation of these results is also shown in Figure 11.Table 8
T-test statistics for questions (13-14, 16-17).
Statement/questiontdfSig.2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersEasy_Modification28.9699403.4633.233.70Easy_Minor_Changes32.0789403.5793.363.80Low_Side_Effects32.1119403.6743.453.90No_Functional_Issues29.8209403.5373.303.77QA LeadsEasy_Modification7.0971103.0832.134.04Easy_Minor_Changes8.8641103.3332.514.16Low_Side_Effects10.661103.5832.844.32No_Functional_Issues9.4661103.5832.754.42TestersEasy_Modification18.6443103.6253.234.02Easy_Minor_Changes20.8793103.7503.384.12Low_Side_Effects19.7743103.8133.424.21No_Functional_Issues16.743103.5943.164.03Figure 11
Histograms for the changeability quality factor from QA lead’s point of view.
(a)(b)(c)(d)


#### 4.1.4. Quick Installability

To access this quality characteristic, questions’ responses (18–20) are subjected to a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-4. The results in Table 9 indicate that the null hypothesis cannot be accepted as significant values are less than 0.05. Therefore, it is concluded that there exists a difference in easy installability for software developed through waterfall and agile. The visual representation of these results is also shown in Figure 12.Table 9
T-test statistics for questions (18–20).
Statement/questiontdfSig. 2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersInstallation_Challenges28.7729403.4953.253.74Installation_Modify31.9989403.6213.403.85Compatibility30.2039403.6213.383.86QA LeadsInstallation_Challenges8.8641103.3332.514.16Installation_Modify10.0001103.3332.604.07Compatibility10.3191103.6672.884.45TestersInstallation_Challenges16.4763103.4693.043.90Installation_Modify24.0763103.8133.494.14Compatibility17.7373103.6883.264.11Figure 12
Histograms for the installability quality factor from QA lead’s point of view.
(a)(b)(c)


#### 4.1.5. Within Time Delivery

To access this quality characteristic, interviews are conducted and subjected to the chi-square test to verify hypothesis-5. The data are collected from 10 software project managers considering the projects completed in the last five years (Table10). The project percentages completed on time through agile and waterfall are 69.23% and 52%, respectively.Table 10
Data received against project managers’ interviews for quality factor delivery on time.
RespondentAgile projectsWaterfall projectsNo. of agile projectsNo. of waterfall projectsCompleted on timeCompleted on time01.0302020202.0701060103.0802060104.0303000205.0204010206.0210010607.0308020308.0202020209.0507040310.04110304Total39502726%Age69.23%52.00%The results in Tables11 and 12 indicate that the chi-square test result is 2.7, which is lower than the critical value of 3.84. Therefore, it is concluded that the null hypothesis is rejected at a 5% significance level. It also highlights no evidence that there is a difference in the timely completion of software projects between waterfall and agile methods.Table 11
Project manager interviews’ observed and expected values for on-time completion of software projects.
ApproachAgileWaterfallTotalSuccessfulObserved272653Expected23.2229.7853UnsuccessfulObserved122436Expected15.7820.2236Total395089Table 12
Chi-square test results for on-time completion of software projects.
OEO−EO−E2O−E2/E2723.223.7814.28840.6153492629.78-3.7814.28840.4797991215.78−3.7814.28840.9054752420.223.7814.28840.706647Statistics from the test2.70727Critical value of chi-square3.841


#### 4.1.6. Within Budget Delivery

To access this quality characteristic, interviews are conducted and subjected to the chi-square test to verify hypothesis-6. The data are collected from 10 software project managers considering the projects completed in the last five years. The project percentages completed within budget with agile and waterfall are 58.97% and 48.00%, respectively.The results in Tables13 and 14 indicate that the chi-square value is 1.05, which is less than critical value as 3.84. Hence, the null hypothesis cannot be rejected at a 5% significance level. It is concluded that there is no evidence to say a difference in the completion of software projects within budget between waterfall and agile methods.Table 13
Statistics of observed/expected values for within budget completion quality characteristics derived from project manager’s responses.
MethodAgileWaterfallTotalSuccessfulObserved232447Expected20.6026.4047UnsuccessfulObserved162642Expected18.4023.6042Total395089Table 14
Chi-square test results for within budget completion of software projects.
OEO−EO−E2O−E2/E2320.602.405.76000.2796122426.40−2.405.76000.2181821618.40−2.405.76000.3130432623.602.405.76000.244068Statistics from the test1.054905Critical value of chi-square3.841


#### 4.1.7. Traditional vs. Agile Methods

A questionnaire was designed with eight questions to compare the agile and traditional software development methodologies. For hypothesis-7, the results in Table15 indicate that significant values are <0.05, so the null hypothesis is rejected at a 5% significance level. Therefore, it is concluded that there is a difference in software projects developed with agile and waterfall.Table 15
T-test statistics–agile vs traditional software development methods’ questions.
Statement/questiontdfSig. 2-tailedMean diff95% confidence interval of diffLowerUpperDevelopersFlexibility26.09403.33.03.6Robustness28.09403.43.23.7Cost effectiveness28.19403.43.23.7Reusability26.29403.33.13.6Reduced risks in success22.79403.12.83.4End user involvement24.69403.22.93.5Delivery on time24.09403.12.83.3Interaction between team members25.29403.02.83.3QA leadsFlexibility8.61103.12.33.9Robustness10.81103.32.74.0Cost effectiveness9.01103.42.64.3Reusability10.21103.42.74.2Reduced risks in success10.01103.62.84.4End user involvement9.81103.22.53.9Delivery on time8.11103.02.23.8Interaction between team members10.21103.42.74.2TestersFlexibility17.53003.83.34.2Robustness19.03003.53.13.9Cost effectiveness16.83003.53.14.0Reusability13.93003.32.83.8Reduced risks in success14.33003.42.93.8End user involvement16.53003.22.83.6Delivery on time12.93003.22.73.7Interaction between team members12.73003.02.53.5


## 4.1. Assessment of Quality Elements

### 4.1.1. Quick Debugging and Correctness

To access this quality characteristic, questions’ responses (5–9) are evaluated using a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-1. The results in Table 6 show that significant values are less than 0.05, so the null hypothesis is not accepted. It indicates that there does exist a difference in faster debugging and correction for the software developed using agile and waterfall. The visual representation is also presented in Figures 8 and 9 from the developers’ point of view.Table 6
T-test statistics for questions (5–9).
Statement/QuestiontdfSig. 2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersUser_Expectation29.6319403.6113.373.85Requirement_Capture34.2319403.7373.523.95System_Design32.6089403.7473.523.98System_Implementation32.2599403.6423.423.87Faults_Free38.6629403.7893.593.98QA LeadsUser_Expectation13.0531104.0833.394.77Requirement_Capture17.111103.9173.414.42System_Design15.6541103.5833.084.09System_Implementation17.8381104.0833.584.59Faults_Free14.1991104.0833.454.72TestersUser_Expectation20.9433103.8133.444.18Requirement_Capture15.7743103.4693.023.92System_Design20.5863103.7813.414.16System_Implementation17.5173103.6563.234.08Faults_Free19.3953103.7193.334.11Figure 8
Histograms for the correctness quality factor from developers’ point of view: (a) user expectation, (b) requirement capture, (c) system design, and (d) system requirement.
(a)(b)(c)(d)Figure 9
Histograms for the correctness quality factor from developers’ “fault free” point of view.


### 4.1.2. Quick Testability

To access this quality characteristic, questions’ responses (10–12,15) are also assessed on a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-2. The results in Table 7 indicate that the null hypothesis is not accepted as significant values are less than 0.05. Therefore, it is concluded that a difference exists in easy testability quality characteristics for software developed with waterfall and agile. The visual representation of these results is also shown in Figure 10.Table 7
T-test statistics for questions (10–12, 15).
Statement/questiontdfSig.2 tailedMean diff95% confidence interval of differenceLowerUpperDevelopersExecution_Test_Script29.6759403.4633.233.69Coding_Standards29.089403.5053.273.74No_Complex_Structure33.0539403.7583.533.98Low_Interaction35.9779403.7583.553.97QA LeadsExecution_Test_Script11.1531103.5002.814.19Coding_Standards17.1101103.9173.414.42No_Complex_Structure10.3831103.5002.764.24Low_Interaction9.7501103.6672.844.49TestersExecution_Test_Script16.5663103.5633.124.00Coding_Standards20.2563103.7503.374.13No_Complex_Structure17.7233103.6253.214.04Low_Interaction16.0913103.6563.194.12Figure 10
Histograms for the testability quality factor from developers’ point of view.
(a)(b)(c)(d)


### 4.1.3. Changeability

To access this quality characteristic, questions’ responses (13–14, 16–17) are evaluated using a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-3. The results in Table 8 indicate that the null hypothesis is not accepted as significant values are less than 0.05. Therefore, it is concluded that there exists a difference in changeability for the software developed with waterfall and agile. The visual representation of these results is also shown in Figure 11.Table 8
T-test statistics for questions (13-14, 16-17).
Statement/questiontdfSig.2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersEasy_Modification28.9699403.4633.233.70Easy_Minor_Changes32.0789403.5793.363.80Low_Side_Effects32.1119403.6743.453.90No_Functional_Issues29.8209403.5373.303.77QA LeadsEasy_Modification7.0971103.0832.134.04Easy_Minor_Changes8.8641103.3332.514.16Low_Side_Effects10.661103.5832.844.32No_Functional_Issues9.4661103.5832.754.42TestersEasy_Modification18.6443103.6253.234.02Easy_Minor_Changes20.8793103.7503.384.12Low_Side_Effects19.7743103.8133.424.21No_Functional_Issues16.743103.5943.164.03Figure 11
Histograms for the changeability quality factor from QA lead’s point of view.
(a)(b)(c)(d)


### 4.1.4. Quick Installability

To access this quality characteristic, questions’ responses (18–20) are subjected to a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-4. The results in Table 9 indicate that the null hypothesis cannot be accepted as significant values are less than 0.05. Therefore, it is concluded that there exists a difference in easy installability for software developed through waterfall and agile. The visual representation of these results is also shown in Figure 12.Table 9
T-test statistics for questions (18–20).
Statement/questiontdfSig. 2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersInstallation_Challenges28.7729403.4953.253.74Installation_Modify31.9989403.6213.403.85Compatibility30.2039403.6213.383.86QA LeadsInstallation_Challenges8.8641103.3332.514.16Installation_Modify10.0001103.3332.604.07Compatibility10.3191103.6672.884.45TestersInstallation_Challenges16.4763103.4693.043.90Installation_Modify24.0763103.8133.494.14Compatibility17.7373103.6883.264.11Figure 12
Histograms for the installability quality factor from QA lead’s point of view.
(a)(b)(c)


### 4.1.5. Within Time Delivery

To access this quality characteristic, interviews are conducted and subjected to the chi-square test to verify hypothesis-5. The data are collected from 10 software project managers considering the projects completed in the last five years (Table10). The project percentages completed on time through agile and waterfall are 69.23% and 52%, respectively.Table 10
Data received against project managers’ interviews for quality factor delivery on time.
RespondentAgile projectsWaterfall projectsNo. of agile projectsNo. of waterfall projectsCompleted on timeCompleted on time01.0302020202.0701060103.0802060104.0303000205.0204010206.0210010607.0308020308.0202020209.0507040310.04110304Total39502726%Age69.23%52.00%The results in Tables11 and 12 indicate that the chi-square test result is 2.7, which is lower than the critical value of 3.84. Therefore, it is concluded that the null hypothesis is rejected at a 5% significance level. It also highlights no evidence that there is a difference in the timely completion of software projects between waterfall and agile methods.Table 11
Project manager interviews’ observed and expected values for on-time completion of software projects.
ApproachAgileWaterfallTotalSuccessfulObserved272653Expected23.2229.7853UnsuccessfulObserved122436Expected15.7820.2236Total395089Table 12
Chi-square test results for on-time completion of software projects.
OEO−EO−E2O−E2/E2723.223.7814.28840.6153492629.78-3.7814.28840.4797991215.78−3.7814.28840.9054752420.223.7814.28840.706647Statistics from the test2.70727Critical value of chi-square3.841


### 4.1.6. Within Budget Delivery

To access this quality characteristic, interviews are conducted and subjected to the chi-square test to verify hypothesis-6. The data are collected from 10 software project managers considering the projects completed in the last five years. The project percentages completed within budget with agile and waterfall are 58.97% and 48.00%, respectively.The results in Tables13 and 14 indicate that the chi-square value is 1.05, which is less than critical value as 3.84. Hence, the null hypothesis cannot be rejected at a 5% significance level. It is concluded that there is no evidence to say a difference in the completion of software projects within budget between waterfall and agile methods.Table 13
Statistics of observed/expected values for within budget completion quality characteristics derived from project manager’s responses.
MethodAgileWaterfallTotalSuccessfulObserved232447Expected20.6026.4047UnsuccessfulObserved162642Expected18.4023.6042Total395089Table 14
Chi-square test results for within budget completion of software projects.
OEO−EO−E2O−E2/E2320.602.405.76000.2796122426.40−2.405.76000.2181821618.40−2.405.76000.3130432623.602.405.76000.244068Statistics from the test1.054905Critical value of chi-square3.841


### 4.1.7. Traditional vs. Agile Methods

A questionnaire was designed with eight questions to compare the agile and traditional software development methodologies. For hypothesis-7, the results in Table15 indicate that significant values are <0.05, so the null hypothesis is rejected at a 5% significance level. Therefore, it is concluded that there is a difference in software projects developed with agile and waterfall.Table 15
T-test statistics–agile vs traditional software development methods’ questions.
Statement/questiontdfSig. 2-tailedMean diff95% confidence interval of diffLowerUpperDevelopersFlexibility26.09403.33.03.6Robustness28.09403.43.23.7Cost effectiveness28.19403.43.23.7Reusability26.29403.33.13.6Reduced risks in success22.79403.12.83.4End user involvement24.69403.22.93.5Delivery on time24.09403.12.83.3Interaction between team members25.29403.02.83.3QA leadsFlexibility8.61103.12.33.9Robustness10.81103.32.74.0Cost effectiveness9.01103.42.64.3Reusability10.21103.42.74.2Reduced risks in success10.01103.62.84.4End user involvement9.81103.22.53.9Delivery on time8.11103.02.23.8Interaction between team members10.21103.42.74.2TestersFlexibility17.53003.83.34.2Robustness19.03003.53.13.9Cost effectiveness16.83003.53.14.0Reusability13.93003.32.83.8Reduced risks in success14.33003.42.93.8End user involvement16.53003.22.83.6Delivery on time12.93003.22.73.7Interaction between team members12.73003.02.53.5


## 4.1.1. Quick Debugging and Correctness

To access this quality characteristic, questions’ responses (5–9) are evaluated using a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-1. The results in Table 6 show that significant values are less than 0.05, so the null hypothesis is not accepted. It indicates that there does exist a difference in faster debugging and correction for the software developed using agile and waterfall. The visual representation is also presented in Figures 8 and 9 from the developers’ point of view.Table 6
T-test statistics for questions (5–9).
Statement/QuestiontdfSig. 2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersUser_Expectation29.6319403.6113.373.85Requirement_Capture34.2319403.7373.523.95System_Design32.6089403.7473.523.98System_Implementation32.2599403.6423.423.87Faults_Free38.6629403.7893.593.98QA LeadsUser_Expectation13.0531104.0833.394.77Requirement_Capture17.111103.9173.414.42System_Design15.6541103.5833.084.09System_Implementation17.8381104.0833.584.59Faults_Free14.1991104.0833.454.72TestersUser_Expectation20.9433103.8133.444.18Requirement_Capture15.7743103.4693.023.92System_Design20.5863103.7813.414.16System_Implementation17.5173103.6563.234.08Faults_Free19.3953103.7193.334.11Figure 8
Histograms for the correctness quality factor from developers’ point of view: (a) user expectation, (b) requirement capture, (c) system design, and (d) system requirement.
(a)(b)(c)(d)Figure 9
Histograms for the correctness quality factor from developers’ “fault free” point of view.


## 4.1.2. Quick Testability

To access this quality characteristic, questions’ responses (10–12,15) are also assessed on a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-2. The results in Table 7 indicate that the null hypothesis is not accepted as significant values are less than 0.05. Therefore, it is concluded that a difference exists in easy testability quality characteristics for software developed with waterfall and agile. The visual representation of these results is also shown in Figure 10.Table 7
T-test statistics for questions (10–12, 15).
Statement/questiontdfSig.2 tailedMean diff95% confidence interval of differenceLowerUpperDevelopersExecution_Test_Script29.6759403.4633.233.69Coding_Standards29.089403.5053.273.74No_Complex_Structure33.0539403.7583.533.98Low_Interaction35.9779403.7583.553.97QA LeadsExecution_Test_Script11.1531103.5002.814.19Coding_Standards17.1101103.9173.414.42No_Complex_Structure10.3831103.5002.764.24Low_Interaction9.7501103.6672.844.49TestersExecution_Test_Script16.5663103.5633.124.00Coding_Standards20.2563103.7503.374.13No_Complex_Structure17.7233103.6253.214.04Low_Interaction16.0913103.6563.194.12Figure 10
Histograms for the testability quality factor from developers’ point of view.
(a)(b)(c)(d)


## 4.1.3. Changeability

To access this quality characteristic, questions’ responses (13–14, 16–17) are evaluated using a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-3. The results in Table 8 indicate that the null hypothesis is not accepted as significant values are less than 0.05. Therefore, it is concluded that there exists a difference in changeability for the software developed with waterfall and agile. The visual representation of these results is also shown in Figure 11.Table 8
T-test statistics for questions (13-14, 16-17).
Statement/questiontdfSig.2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersEasy_Modification28.9699403.4633.233.70Easy_Minor_Changes32.0789403.5793.363.80Low_Side_Effects32.1119403.6743.453.90No_Functional_Issues29.8209403.5373.303.77QA LeadsEasy_Modification7.0971103.0832.134.04Easy_Minor_Changes8.8641103.3332.514.16Low_Side_Effects10.661103.5832.844.32No_Functional_Issues9.4661103.5832.754.42TestersEasy_Modification18.6443103.6253.234.02Easy_Minor_Changes20.8793103.7503.384.12Low_Side_Effects19.7743103.8133.424.21No_Functional_Issues16.743103.5943.164.03Figure 11
Histograms for the changeability quality factor from QA lead’s point of view.
(a)(b)(c)(d)


## 4.1.4. Quick Installability

To access this quality characteristic, questions’ responses (18–20) are subjected to a 1-sampleT-test of software developers, QA leads, and software testers, and associated with hypothesis-4. The results in Table 9 indicate that the null hypothesis cannot be accepted as significant values are less than 0.05. Therefore, it is concluded that there exists a difference in easy installability for software developed through waterfall and agile. The visual representation of these results is also shown in Figure 12.Table 9
T-test statistics for questions (18–20).
Statement/questiontdfSig. 2-tailedMean diff95% confidence interval of differenceLowerUpperDevelopersInstallation_Challenges28.7729403.4953.253.74Installation_Modify31.9989403.6213.403.85Compatibility30.2039403.6213.383.86QA LeadsInstallation_Challenges8.8641103.3332.514.16Installation_Modify10.0001103.3332.604.07Compatibility10.3191103.6672.884.45TestersInstallation_Challenges16.4763103.4693.043.90Installation_Modify24.0763103.8133.494.14Compatibility17.7373103.6883.264.11Figure 12
Histograms for the installability quality factor from QA lead’s point of view.
(a)(b)(c)


## 4.1.5. Within Time Delivery

To access this quality characteristic, interviews are conducted and subjected to the chi-square test to verify hypothesis-5. The data are collected from 10 software project managers considering the projects completed in the last five years (Table10). The project percentages completed on time through agile and waterfall are 69.23% and 52%, respectively.Table 10
Data received against project managers’ interviews for quality factor delivery on time.
RespondentAgile projectsWaterfall projectsNo. of agile projectsNo. of waterfall projectsCompleted on timeCompleted on time01.0302020202.0701060103.0802060104.0303000205.0204010206.0210010607.0308020308.0202020209.0507040310.04110304Total39502726%Age69.23%52.00%The results in Tables11 and 12 indicate that the chi-square test result is 2.7, which is lower than the critical value of 3.84. Therefore, it is concluded that the null hypothesis is rejected at a 5% significance level. It also highlights no evidence that there is a difference in the timely completion of software projects between waterfall and agile methods.Table 11
Project manager interviews’ observed and expected values for on-time completion of software projects.
ApproachAgileWaterfallTotalSuccessfulObserved272653Expected23.2229.7853UnsuccessfulObserved122436Expected15.7820.2236Total395089Table 12
Chi-square test results for on-time completion of software projects.
OEO−EO−E2O−E2/E2723.223.7814.28840.6153492629.78-3.7814.28840.4797991215.78−3.7814.28840.9054752420.223.7814.28840.706647Statistics from the test2.70727Critical value of chi-square3.841


## 4.1.6. Within Budget Delivery

To access this quality characteristic, interviews are conducted and subjected to the chi-square test to verify hypothesis-6. The data are collected from 10 software project managers considering the projects completed in the last five years. The project percentages completed within budget with agile and waterfall are 58.97% and 48.00%, respectively.The results in Tables13 and 14 indicate that the chi-square value is 1.05, which is less than critical value as 3.84. Hence, the null hypothesis cannot be rejected at a 5% significance level. It is concluded that there is no evidence to say a difference in the completion of software projects within budget between waterfall and agile methods.Table 13
Statistics of observed/expected values for within budget completion quality characteristics derived from project manager’s responses.
MethodAgileWaterfallTotalSuccessfulObserved232447Expected20.6026.4047UnsuccessfulObserved162642Expected18.4023.6042Total395089Table 14
Chi-square test results for within budget completion of software projects.
OEO−EO−E2O−E2/E2320.602.405.76000.2796122426.40−2.405.76000.2181821618.40−2.405.76000.3130432623.602.405.76000.244068Statistics from the test1.054905Critical value of chi-square3.841


## 4.1.7. Traditional vs. Agile Methods

A questionnaire was designed with eight questions to compare the agile and traditional software development methodologies. For hypothesis-7, the results in Table15 indicate that significant values are <0.05, so the null hypothesis is rejected at a 5% significance level. Therefore, it is concluded that there is a difference in software projects developed with agile and waterfall.Table 15
T-test statistics–agile vs traditional software development methods’ questions.
Statement/questiontdfSig. 2-tailedMean diff95% confidence interval of diffLowerUpperDevelopersFlexibility26.09403.33.03.6Robustness28.09403.43.23.7Cost effectiveness28.19403.43.23.7Reusability26.29403.33.13.6Reduced risks in success22.79403.12.83.4End user involvement24.69403.22.93.5Delivery on time24.09403.12.83.3Interaction between team members25.29403.02.83.3QA leadsFlexibility8.61103.12.33.9Robustness10.81103.32.74.0Cost effectiveness9.01103.42.64.3Reusability10.21103.42.74.2Reduced risks in success10.01103.62.84.4End user involvement9.81103.22.53.9Delivery on time8.11103.02.23.8Interaction between team members10.21103.42.74.2TestersFlexibility17.53003.83.34.2Robustness19.03003.53.13.9Cost effectiveness16.83003.53.14.0Reusability13.93003.32.83.8Reduced risks in success14.33003.42.93.8End user involvement16.53003.22.83.6Delivery on time12.93003.22.73.7Interaction between team members12.73003.02.53.5


## 5. Threats to Validity

There are some limitations to this work. First, this case study focuses on the quality characteristics related to the software developers. Second, data are gathered to validate whether there is a contrast in completing the project within time and budget using software development process techniques, i.e., agile and waterfall. Third, we have not separated results based on the size of the project because of the data accumulation problem within the specified period. Last, the study was carried out in Pakistan and gathered data from public sector software development organizations. Hence, the results speak about the experiences of IT experts and professionals, particularly in public sector organizations in Pakistan; however, other software development organizations in Pakistan and around the globe were not considered.


## 6. Conclusion and Future Work

The software industry faces major challenges, e.g., high software quality, ever-increasing software complexity, low budgets, and tight schedules. To address these issues, agile software development approaches provide an effective solution. Adopting agile methods across the private sector has proven their usefulness and effectiveness. However, it remains questionable for the public sector. Our study intends to contribute to the already existing body of knowledge about the usefulness of agile methodologies for the public sector environment to improve the software quality. Our questionnaire-based survey results demonstrate a clear trend toward the effectiveness of agile-based development compared to traditional software methodologies. For quality factors, i.e., “quick debugging and correctness,” “maintenance,” “easy testing,” and “installation,” agile methods have distinct superiority over traditional software development for software development companies in a public sector environment. The results suggest that agile development techniques or methods must be used for faster delivery, easy debugging, easy testability, and easy installability, in a public software development organization. In the future, other project quality components will be considered, such as time, cost, and the most suitable strategy between agile and waterfall.



---

*Source: 6153744-2022-08-29.xml*