# Analysis of Environmental Factors for Mobile Software Development Focused on Korean Companies

Authors: Kwang Yoon Song; In Hong Chang; Hoang Pham
Journal: Mobile Information Systems
 (2021)

---

## Abstract

Software should be a product that can be used easily and accurately by the user and should be improved quickly and accurately when problems arise. In many software development projects, software requirements are frequently modified during the design or development phase, which tests the development of specific designers’ or developers’ capabilities. Software development environmental factors (SDEFs), such as differences in mutual work recognition among users, developers, and testers or knowledge differences, can hinder communication, which may lead to faulty development owing to erroneous job definition. Because the exact size and scope of the software cannot be calculated, the risk of excessive requirements, such as schedule, cost, and manpower, may increase. This study aims to investigate 32 SDEFs to examine the influence of factors affecting the reliability of software developed by Korean companies to identify factors with high influence and compare differences with previous studies. Moreover, we found whether any new SDEFs from the top 10 rankings only affected Korean companies, and also US companies included in previous studies. A factor analysis revealed several potential factors to identify the mutually independent characteristics of the factors. Through statistical analysis methods, the difference between the group means and the impact on improving the software reliability were found in Korean companies. These findings can provide useful benefits to software developers and managers working in countries with different or similar cultures and help increase working efficiency, i.e., work versus time investment and software reliability improvement.


---

## Body

## 1. Introduction

When using software for personal use, the schedule or quality of development is not a concern. However, good software that is meant for sale to the general public should be user-friendly, error-free, efficient, reliable, and utilitarian to satisfy its users. Software systems that offer user convenience must improve solutions for immediate problems in multiple industries. Software development becomes a more complex process as the requirements for system quality increase. With more elaborate software projects and pressing deadlines, software applications have also become larger and more integrated. Projects and their applications require interaction between diverse individuals in different roles to deliver projects on time, within budget, and while adhering to feature and functional requirements [1]. Software testing is indispensable to ensure quality, wherein critical contributions to strategic objectives set by the firm are met by the information technology (IT) unit [2]. However, the continuing mismatch among the goals of testers, developers, and the IT unit is large, and threatens the firm’s corporate success [3]. Paternoster et al. [4] conducted a systematic mapping study by developing a classification schema, ranking the selected primary studies according to their rigor and relevance and analyzing the reported software development work practices in start-ups. Turk et al. [5] explained the integrated agile development environment to identify underlying assumptions and limitations. Currently, many software companies are working hard to match the advantages of open-source software.


### 1.1. Motivation

As the era of the 4th Industrial Revolution arrives, the importance of software is increasing day by day, and it is a field of interest not only for global companies but also for all countries. Recently, many companies and countries have been investing in various fields to focus on software development. In addition, noting the importance of investing in research and development (R&D), the Organization for Economic Co-operation and Development (OECD) countries and other nations have been increasing support for R&D as part of their science, technology, and industrial policy. As technological development relies on R&D, governments have been leveraging institutions and policies to expand such investments in the public and private sectors to achieve qualitative improvements. In Table1, looking at the gross domestic product (GDP) by country in 2019, the GDP is 21.4, 14.3, 5.1, 3.9, and 2.9 trillion dollars for the USA, China, Japan, Germany, and India, respectively. In comparison, Korea’s GDP is relatively small at 1.6 trillion dollars. Moreover, looking at the GDP by country in 2018, the GDP is 20.6, 13.9, 5.0, 4.0, and 2.9 trillion dollars for the USA, China, Japan, Germany, and UK, respectively. In comparison, Korea’s GDP is again relatively small at 1.7 trillion dollars [6]. Table 2 shows the gross domestic spending on R&D [7]. It shows that the US has higher R&D investment than Korea and Japan, but their investment ratio to GDP is the lowest. Korea has a lower total investment than the US and Japan, but the rate of investment and growth relative to GDP has been rising rapidly [8]. Many countries are making efforts to increase R&D investments in the private and public sectors.Table 1
GDP status of the world’s top 12 countries.
No.Country2017 (millions)2018 (millions)2019 (millions)1United States19,519,353.6920,580,159.7821,433,226.002China12,310,409.3713,894,817.5514,342,903.013Japan4,866,864.414,954,806.625,081,769.544Germany3,682,602.483,963,767.533,861,123.565India2,652,754.692,713,165.062,868,929.426United Kingdom2,666,229.182,860,667.732,829,108.227France2,595,151.052,787,863.962,715,518.278Italy2,062,831.052,091,544.962,003,576.159Brazil1,961,796.201,885,482.531,839,758.0410Canada1,649,878.051,716,262.621,736,425.6311Russia1,574,199.391,669,583.091,699,876.5812Korea1,623,901.501,724,845.621,646,739.22Table 2
Gross domestic spending on R&D of the world’s top 10 countries.
No.Country201620172018PricePercentPricePercentPricePercent1United States511,296.72.76533,313.12.81551,517.72.832China398,951.62.10429,092.42.12462,577.62.143Japan163,035.03.16169,213.43.21173,313.43.284Germany117,109.82.94125,175.53.07129,647.13.135Korea79,375.43.9988,147.74.29954,61.74.536France61,093.62.2261,962.02.20627,71.32.197United Kingdom46,847.21.6648,308.31.68503,73.01.738Chinese Taipei35,398.53.1537,975.53.28411,04.73.469Russia38,947.81.1039,921.01.11362,51.80.9810Canada27,847.61.7327,735.51.67264,97.11.56Apple has once again risen to be the world’s largest technology company in 2020, according to the Forbes magazine. Despite Korea’s small GDP, there are relatively large number of Korean companies on the world’s top companies list, such as Samsung Hyundai Motor, SK Hynix, POSCO, and LG Electronics. This has been made possible not just because of hardware technology but because companies realized the importance of software development and considered the factors affecting the software development process. Samsung is one such example. Samsung’s first smartphone saw a tremendous improvement in terms of hardware. However, when it was launched in an awkward fashion, it had the disadvantage of being the worst smartphone because of software optimization issues. It is important to ensure natural usability through software. Samsung is also making attempts to develop new software to lead the global smartphone industry. In addition, generally, we encountered various problems in software development. To overcome the problems arising in the process of software development, developers have used various approaches [9], and the development of communication technology is one such approach. In this study, we investigate the cultural differences, human cooperation, and tactics aimed at reducing the temporal distance and not the technical aspects. Considering the current status of the mobile content industry, the global app store consumer spending in 2019 was $120 billion, an increase of 2.1 times compared to 2016, and mature markets such as the US, Japan, Korea, and the UK have been driving the growth of app store consumer spending. In addition, the main factors include increased consumer awareness in the mobile industry, increased mobile device users and time spent on mobile devices, advertising revenue from apps, increased influence of mobile games, and increased in-app spending based on subscriptions. Games account for 72% of the app store consumer spending, while subscription services drive the spending growth for nongame apps. In 2019, the number of global app store app downloads (Google Play) was 204 billion, an increase of 45% compared to 2016 and an increase of 6% compared to the previous year. The number of mobile subscribers worldwide is expected to increase by 1.9% per year from 5.1 billion in 2018 to 5.8 billion by 2025, and the growth of the mobile industry is expected to continue until 2025 [10].


### 1.2. Research Objectives

Alexy et al. [11] found that the support for organizational innovation that allows commercial engagement with open-source software has varying impacts on the technical and administrative dimensions of different job roles through a mixed-method research design. They found that individual-level attributes can counterbalance job role changes that weaken support for adopting open-source software, while perceived organizational commitment has no effect. Ropponen and Lyytinen [12] examined how risk management and environmental factors, such as the development methods and managers’ experience, influence risk components. Their analysis results show that awareness of the importance of risk management and systematic practices to manage risks has an influence on risks related to scheduling, requirements management, and personnel management. Zhang and Pham [13] defined software development environmental factors (SDEFs) related to the software development process. In addition, the study conducted a survey to investigate rankings based on the factors’ influence on software reliability assessment, and also conducted exploratory analyses and presented various opinions about improvement of software reliability. Opinions were presented through additional investigations and analysis results to examine the changes made by software development practitioners. In many software development projects, software requirements are frequently modified during design or development, which tests the development of specific designers’ or developers’ capabilities. SDEFs, such as differences in mutual work recognition among users, developers, and testers, or knowledge differences, can hinder communication, which may lead to faulty development owing to erroneous job definition. Because the exact size and scope of the software cannot be calculated, the risk of excessive requirements, such as schedule, cost, and manpower, may increase.The objectives of this study are as follows:(1)
This study investigates 32 SDEFs to examine the influence of factors affecting the software reliability of Korean companies to identify factors with high influence.(2)
We find the most important SDEFs for Korean companies and find the correlations between specific SDEFs.(3)
A factor analysis is conducted to identify the mutually independent characteristics of the factors by finding several potential (correlated) factors in the SDEFs, and the differences in group means are investigated through analysis of variance (ANOVA) in the statistical model. This study also uses regression analysis to describe the impact of configuration on improving software stability and analyzes the stages of the development lifecycle.(4)
Finally, we compare the results with previous studies to find the characteristics that only affect Korean companies and discuss the causes of the differences.Section2 presents the details of the data collected. Section 3 presents the results of various analyses (factor, correlation, regression analysis, etc.) to examine the diversity of Korean companies. Section 4 compares the new findings related to Korean companies with the results of previous studies. It also explains the causes for the existence of differences and similarities in software development. Section 5 concludes the study and discusses the differences.


## 1.1. Motivation

As the era of the 4th Industrial Revolution arrives, the importance of software is increasing day by day, and it is a field of interest not only for global companies but also for all countries. Recently, many companies and countries have been investing in various fields to focus on software development. In addition, noting the importance of investing in research and development (R&D), the Organization for Economic Co-operation and Development (OECD) countries and other nations have been increasing support for R&D as part of their science, technology, and industrial policy. As technological development relies on R&D, governments have been leveraging institutions and policies to expand such investments in the public and private sectors to achieve qualitative improvements. In Table1, looking at the gross domestic product (GDP) by country in 2019, the GDP is 21.4, 14.3, 5.1, 3.9, and 2.9 trillion dollars for the USA, China, Japan, Germany, and India, respectively. In comparison, Korea’s GDP is relatively small at 1.6 trillion dollars. Moreover, looking at the GDP by country in 2018, the GDP is 20.6, 13.9, 5.0, 4.0, and 2.9 trillion dollars for the USA, China, Japan, Germany, and UK, respectively. In comparison, Korea’s GDP is again relatively small at 1.7 trillion dollars [6]. Table 2 shows the gross domestic spending on R&D [7]. It shows that the US has higher R&D investment than Korea and Japan, but their investment ratio to GDP is the lowest. Korea has a lower total investment than the US and Japan, but the rate of investment and growth relative to GDP has been rising rapidly [8]. Many countries are making efforts to increase R&D investments in the private and public sectors.Table 1
GDP status of the world’s top 12 countries.
No.Country2017 (millions)2018 (millions)2019 (millions)1United States19,519,353.6920,580,159.7821,433,226.002China12,310,409.3713,894,817.5514,342,903.013Japan4,866,864.414,954,806.625,081,769.544Germany3,682,602.483,963,767.533,861,123.565India2,652,754.692,713,165.062,868,929.426United Kingdom2,666,229.182,860,667.732,829,108.227France2,595,151.052,787,863.962,715,518.278Italy2,062,831.052,091,544.962,003,576.159Brazil1,961,796.201,885,482.531,839,758.0410Canada1,649,878.051,716,262.621,736,425.6311Russia1,574,199.391,669,583.091,699,876.5812Korea1,623,901.501,724,845.621,646,739.22Table 2
Gross domestic spending on R&D of the world’s top 10 countries.
No.Country201620172018PricePercentPricePercentPricePercent1United States511,296.72.76533,313.12.81551,517.72.832China398,951.62.10429,092.42.12462,577.62.143Japan163,035.03.16169,213.43.21173,313.43.284Germany117,109.82.94125,175.53.07129,647.13.135Korea79,375.43.9988,147.74.29954,61.74.536France61,093.62.2261,962.02.20627,71.32.197United Kingdom46,847.21.6648,308.31.68503,73.01.738Chinese Taipei35,398.53.1537,975.53.28411,04.73.469Russia38,947.81.1039,921.01.11362,51.80.9810Canada27,847.61.7327,735.51.67264,97.11.56Apple has once again risen to be the world’s largest technology company in 2020, according to the Forbes magazine. Despite Korea’s small GDP, there are relatively large number of Korean companies on the world’s top companies list, such as Samsung Hyundai Motor, SK Hynix, POSCO, and LG Electronics. This has been made possible not just because of hardware technology but because companies realized the importance of software development and considered the factors affecting the software development process. Samsung is one such example. Samsung’s first smartphone saw a tremendous improvement in terms of hardware. However, when it was launched in an awkward fashion, it had the disadvantage of being the worst smartphone because of software optimization issues. It is important to ensure natural usability through software. Samsung is also making attempts to develop new software to lead the global smartphone industry. In addition, generally, we encountered various problems in software development. To overcome the problems arising in the process of software development, developers have used various approaches [9], and the development of communication technology is one such approach. In this study, we investigate the cultural differences, human cooperation, and tactics aimed at reducing the temporal distance and not the technical aspects. Considering the current status of the mobile content industry, the global app store consumer spending in 2019 was $120 billion, an increase of 2.1 times compared to 2016, and mature markets such as the US, Japan, Korea, and the UK have been driving the growth of app store consumer spending. In addition, the main factors include increased consumer awareness in the mobile industry, increased mobile device users and time spent on mobile devices, advertising revenue from apps, increased influence of mobile games, and increased in-app spending based on subscriptions. Games account for 72% of the app store consumer spending, while subscription services drive the spending growth for nongame apps. In 2019, the number of global app store app downloads (Google Play) was 204 billion, an increase of 45% compared to 2016 and an increase of 6% compared to the previous year. The number of mobile subscribers worldwide is expected to increase by 1.9% per year from 5.1 billion in 2018 to 5.8 billion by 2025, and the growth of the mobile industry is expected to continue until 2025 [10].


## 1.2. Research Objectives

Alexy et al. [11] found that the support for organizational innovation that allows commercial engagement with open-source software has varying impacts on the technical and administrative dimensions of different job roles through a mixed-method research design. They found that individual-level attributes can counterbalance job role changes that weaken support for adopting open-source software, while perceived organizational commitment has no effect. Ropponen and Lyytinen [12] examined how risk management and environmental factors, such as the development methods and managers’ experience, influence risk components. Their analysis results show that awareness of the importance of risk management and systematic practices to manage risks has an influence on risks related to scheduling, requirements management, and personnel management. Zhang and Pham [13] defined software development environmental factors (SDEFs) related to the software development process. In addition, the study conducted a survey to investigate rankings based on the factors’ influence on software reliability assessment, and also conducted exploratory analyses and presented various opinions about improvement of software reliability. Opinions were presented through additional investigations and analysis results to examine the changes made by software development practitioners. In many software development projects, software requirements are frequently modified during design or development, which tests the development of specific designers’ or developers’ capabilities. SDEFs, such as differences in mutual work recognition among users, developers, and testers, or knowledge differences, can hinder communication, which may lead to faulty development owing to erroneous job definition. Because the exact size and scope of the software cannot be calculated, the risk of excessive requirements, such as schedule, cost, and manpower, may increase.The objectives of this study are as follows:(1)
This study investigates 32 SDEFs to examine the influence of factors affecting the software reliability of Korean companies to identify factors with high influence.(2)
We find the most important SDEFs for Korean companies and find the correlations between specific SDEFs.(3)
A factor analysis is conducted to identify the mutually independent characteristics of the factors by finding several potential (correlated) factors in the SDEFs, and the differences in group means are investigated through analysis of variance (ANOVA) in the statistical model. This study also uses regression analysis to describe the impact of configuration on improving software stability and analyzes the stages of the development lifecycle.(4)
Finally, we compare the results with previous studies to find the characteristics that only affect Korean companies and discuss the causes of the differences.Section2 presents the details of the data collected. Section 3 presents the results of various analyses (factor, correlation, regression analysis, etc.) to examine the diversity of Korean companies. Section 4 compares the new findings related to Korean companies with the results of previous studies. It also explains the causes for the existence of differences and similarities in software development. Section 5 concludes the study and discusses the differences.


## 2. Data Collection

Likert [14] introduced the Likert scale and technique, wherein an individual is invited to define his attitude toward each statement by choosing among a number of r grades on the r-grade Likert scale. The most popular are the five-grade and seven-grade Likert scales. We used the survey by Zhang and Pham [13], which included 32 SDEFs and background information of the software developer. In the survey form [13], 1 indicates “not important,” i.e., it has almost no effect. Conversely, 7 indicates “most important,” which implies it has a significant impact on software reliability. The data were collected from a formal survey and Internet (e-mail) survey served directly to the managers, system engineers, and software developers of 11 organizations (mobile software development, general software development companies etc.) in Korea; 89 survey responses were obtained, among which, 75 were used, excluding a few surveys (written incorrectly or where all values are the same), which were deemed unfit for analysis. R, SPSS V.24, and Excel 2014 were used for the analysis. Table 3 summarizes the demographic data of the participants. The sample had a relatively good mixture of software development experience, size diversity, and program categories, and also represented a good mixture of software development forms. The average number of years of software development was 8.41 and the average percentage of reused code was 49.13.Table 3
Demographic data.
Demographic factorSubfactorSample sizePercentApplication developmentSafety-critical1013.3Commercial2229.3Inside user-oriented3242.7Other1114.7Total75100.0Type of software development experienceDatabase1317.3Operation system810.7Communication control1114.7Language processor1621.3Other2736.0Total75100.0Title/PositionManager1114.7System engineer1013.3Programmer2229.3Tester1114.7Administrator1114.7Other1013.3Total75100.0Number of years of software developmentLess than 5 years2938.76–10 years2229.3More than 11 years2432.0Total75100.0Percentage of time spent (%)Analysis7521.33Design7525.33Coding7530.40Testing7522.93Total100.0Average of number of years of software development (years)758.41Reused code (%)7549.13


## 3. Results

### 3.1. Relative Importance of SDEFs

#### 3.1.1. Importance of SDEFs Based on the Relative Weighted Method

The survey consisted of 32 SDEFs, with 7 points representing the most important factor in software reliability and 0 points indicating the least important. The relative weighting method with each score was applied to determine the ranking [13]. Table 4 shows the results of analyzing the ranking of 32 software development SDEFs using the relative weighting method. Typically, an environmental factor with a higher normalized weight has a more significant impact on the software reliability assessment than the factor that has a lower weight. Therefore, software developers should pay more attention to SDEFs with higher normalized weights. As can be seen in Table 4, in Korean companies, the normalized weight value of factor f11 (requirement analysis) is the highest at 0.03667, followed by f15 (programmer skill) at 0.03634, f12 (relationship of detailed design to requirement) at 0.03578, f20 (human nature) at 0.03449, f22 (testing effort) at 0.03449, and f24 (testing methodologies) at 0.03449.Table 4
The results of normalized weighted ranking.
RankFactorFactor nameNormalized weight1f11Requirements analysis0.036672f15Programmer skill0.036343f12Relationship of detailed design to requirement0.035784f20Human nature0.034495f22Testing effort0.034496f24Testing methodologies0.034497f18Program workload (stress)0.034328f6Percentage of reused modules0.034249f19Domain knowledge0.0340810f7Programming language0.0340011f25Testing coverage0.0338412f2Program categories0.0337613f13Work standards0.0336814f5Level of programming technologies0.0335215f14Development management0.0334416f8Frequency of program specification change0.0333517f1Program complexity0.0328718f4Amount of programming effort0.0328719f21Testing environment0.0322220f3Difficulty of programming0.0318221f9Volume of program design documents0.0312622f10Design methodology0.0309323f17Development team size0.0302124f23Testing resource allocation0.0297225f27Documentation0.0296426f26Testing tools0.0288327f29Storage devices0.0255228f32System software0.0243129f30Input/output devices0.0235830f28Processors0.0222931f31Telecommunication devices0.0222132f16Programmer organization0.02124


#### 3.1.2. Correlation Analysis between SDEFs

This section describes a correlation analysis performed to examine the correlation between SDEFs. The Pearson correlation coefficient (Pearson’sr) is a measure of the linear dependence between two variables, where 1 is the total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation [15]. Table 5 shows the correlation of the top-10 SDEFs with high normalized weight values in Table 4. Correlation analysis was performed using 32 SDEFs; only the SDEFs that were significant in the 10 SDEFs shown in Table 5 were tabulated and the other SDEFs were excluded, and only those factors with a strong correlation were summarized (r > 0.4 or r < −0.4). As a fragmentary example, f11 (requirements analysis) and f12 (relationship of detailed design to requirement), and f11 (requirements analysis) and f13 (work standards) have a strong quantitative linear relationship.Table 5
Correlation analysis between SDEFs.
Factor (factor name)Correlated factor (factor name)Pearson’srf11 (requirements analysis)f12 (relationship of detailed design to requirement)0.656∗∗f13 (work standards)0.598∗∗f10 (design methodology)0.541∗∗f17 (development team size)0.491∗∗f14 (development management)0.464∗∗f18 (program workload (stress))0.451∗∗f6 (percentage of reused modules)0.421∗∗f9 (volume of program design documents)0.412∗∗f25 (testing coverage)0.400∗∗f15 (programmer skill)f19 (domain knowledge)0.629∗∗f3 (difficulty of programming)0.533∗∗f20 (human nature)0.423∗∗f12 (relationship of detailed design to requirement)f11 (requirements analysis)0.656∗∗f10 (design methodology)0.406∗∗f20 (human nature)f7 (programming language)0.453∗∗f14 (development management)0.439∗∗f15 (programmer skill)0.423∗∗f16 (programmer organization)0.407∗∗f22 (testing effort)f25 (testing coverage)0.741∗∗f24 (testing methodologies)0.728∗∗f27 (documentation)0.438∗∗f28 (processors)−0.419∗∗f25 (testing methodologies)f25 (testing coverage)0.843∗∗F22 (testing effort)0.728∗∗f27 (documentation)0.659∗∗F26 (testing tools)0.503∗∗f23 (testing resource allocation)0.420∗∗f28 (processors)−0.419∗∗f18 (program workload (stress))f14 (development management)0.453∗∗f11 (requirements analysis)0.451∗∗f5 (level of programming technologies)0.449∗∗f13 (work standards)0.414∗∗f6 (percentage of reused modules)f11 (requirements analysis)0.421∗∗f10 (design methodology)0.414∗∗f19 (domain knowledge)F15 (programmer skill)0.629∗∗f3 (difficulty of programming)0.426∗∗f5 (level of programming technologies)0.426∗∗f7 (programming language)f20 (human nature)0.453∗∗f14 (development management)0.411∗∗f17 (development team size)0.403∗∗∗∗p<0.01.


### 3.2. Environmental Factor Analysis

#### 3.2.1. Factor Analysis

Factor analysis was conducted to verify the construct validity of the questionnaire. Construct validity is determined by the suitability of the items included in the questionnaire to evaluate the set of hypothesized constructs. Factorial validity is a construct validity that proves the validity through factor analysis. Therefore, factor analysis was conducted to verify the validity of the items related to the improvement of the software reliability assessment [16]. Factor analysis was performed using the principal component method, and the varimax method was used as the orthogonal rotation method. The eigenvalues of the components, proportion of variance, and cumulative proportion of variance are presented in Table 6, and 70.973% of the variation can be explained by the first four components. The first to fourth eigenvalues are greater than 1; however, the value after the fifth eigenvalue is less than 1. Those with eigenvalues less than 1 are not considered to be stable. They account for less variability than a single variable does, and hence, they are not retained in the analysis. In this sense, we end up with fewer factors than the original number of variables [17]; hence, the first four components are retained. As a result, the selected factors are equal to the top 13 most important SDEFs based on the ranking results presented by the relative weighting method in Table 4. We can observe that the SDEFs cover many aspects of the development and testing of software, and the majority of these factors have a similarly significant impact on software reliability.Table 6
Eigenvalue of the correlation matrix.
ComponentsEigenvalueProportion of varianceCumulative proportion14.04131.08231.08222.68220.63151.71331.39310.71762.43041.1118.54370.97350.8356.42077.39360.7305.61883.01170.5063.89686.90780.4993.84190.74790.3592.75893.505100.3002.31095.814110.2652.04197.856120.1491.14799.003130.1300.997100.000As shown in Table7, we retained four principal components: PCA1, PCA2, PCA3, and PCA4. The first component, PCA1, consisted of factors for f24, f25, and f22, which are related to testing ability, whereas PCA2 consists of factors for f15, f19, f20, f7, and f18, which are related to programmer ability. PCA3 consisted of factors for f11, f12, and f13, whereas PCA4 consisted of factors f2 and f6, which were related to requirements, standards, and program utility, respectively. All four components had high loading values, which exceeded 0.5. Several methods have been proposed as a measure of reliability, but in this study, Cronbach’s α test was used to test the internal consistency of the items. Cronbach’s α typically increases as the intercorrelations among test items increase and is, thus, known as an internal consistency estimate of the reliability of test scores. If the Cronbach’s α coefficient is more than 0.6, the scale is reliable [18]. As can be seen in Table 7, Cronbach’s α is 0.786, which is greater than 0.6.Table 7
New principles for each principal component.
ComponentNew principleFactor (name)LoadingRotation sums of squared loadingsCronbach’sαEigenvalueProportion of varianceCumulative proportionPCA1Testing abilityf24 (testing methodologies)0.8982.76421.26221.2620.786f25 (testing coverage)0.895f22 (testing effort)0.857PCA2Programmer abilityf15 (programmer skill)0.8052.49919.22540.486f19 (domain knowledge)0.731f20 (human nature)0.718f7 (programming language)0.618f18 (program workload)0.531PCA3Requirements and standardsf11 (requirements analysis)0.8712.20216.94157.427f12 (relationship of detailed design to requirement)0.827f13 (work standards)0.642PCA4Program utilityf2 (program categories)0.7491.76113.54570.973f6 (percentage of reused modules)0.682


#### 3.2.2. Hypothesis Test

A statistical hypothesis is a method of statistical interference that is testable based on observing a process that is modeled via a set of random variables [19]. Hypothesis tests are used to determine the outcomes of a study that would lead to a rejection of the null hypothesis for a pre-specified level of significance. The process of distinguishing between the null hypothesis and the alternative hypothesis is aided by identifying two conceptual types of errors, such as type 1 and type 2, and by specifying parametric limits on, for instance, how much type 1 error will be permitted. The null hypothesis H0 is a simple hypothesis related to the contradiction, with the theory to be proved. The alternative hypothesis H1 is a hypothesis related to the theory to be proved. The hypothesis will reveal whether their opinion can be considered “the same.”Hypothesis 1.
People developing software for different applications have the same opinion on the importance of the four components of the new principle. Safety-critical, commercial, inside-used systems, and others may not have the same opinion on the significance of all these factors.(1)H0:μs=μc=μi=μo,H1:atleast onμiis different.Hypothesis 2.
People with different types of software development experience have the same opinion on the significance of the four components of the new principle. The database, operation system, communication control, language processor, and others are considered.(2)H0:μd=μo=μc=μl=μo,H1:atleast onμiis different.Hypothesis 3.
People playing different numbers of years of software development have the same opinion on the significance of the four components of the new principle. Less than 5 years, 6–10 years, and more than 11 years are considered here.(3)H0:μ≤5=μ6−10=μ≥11,H1:atleast onμiis different.Hypothesis 4.
People playing different roles in software development have the same opinion on the significance of the four components of the new principle. Managers, system engineers, programmers, testers, administrators, and others are considered here.(4)H0:μm=μs=μp=μt=μa=μo,H1:atleast onμiis different.Hypothesis 5.
People playing different numbers of years of software development have the same opinion on the significance of the percentage of reused code. Less than 5 years, 6–10 years, and more than 11 years are considered here.(5)H0:μ≤5=μ6−10=μ≥11,H1:atleast onμiis different.
We performed ANOVA on hypotheses 1 to 5. ANOVAs are useful for comparing three or more means (groups or variables) for statistical significance. ANOVA is suitable for a wide range of practical problems. The results of the analysis for the given hypotheses are presented in Tables8–12. As per the results of Hypothesis 1, there was no significant difference according to the group, as shown in Table 8. As per the results of Hypothesis 2, there is a significant difference only in the group of program utility component factors, as shown in Table 9. A post-hoc analysis was performed using Scheffe’s test method; however, there was no significant difference between the groups. As per the results of Hypothesis 3, there is a significant difference only in the programmer ability component factor group, as shown in Table 10. In the post-hoc analysis, there was a significant difference between groups of less than 5 years and more than 11 years. As per the results of Hypothesis 4, there is a significant difference only in the group of program utility component factors, as shown in Table 11. Post-hoc analysis showed no significant differences between the groups. According to the results of Hypothesis 5, there is a significant difference in the percentage of reused code, as shown in Table 12. In the post-hoc analysis, there was a significant difference between groups of less than 5 years and 5–10 years, and less than 5 years and more than 11 years. In Tables 8–12, N is the frequency, M is the mean value, SD is the standard deviation, F is the variance between groups indicating the degree of difference between groups, and P is the p value. Figures 1–5 show the results of ANOVA for Hypothesis 1–5.Table 8
Results of ANOVA for Hypothesis1.
New principleDemographic factorNMSDFPTesting abilitySafety-critical105.8331.0330.4520.716Commercial225.8181.246Inside user-oriented325.5311.040Other115.5450.637Total755.6581.048Programmer abilitySafety-critical105.4800.8180.3920.759Commercial225.7910.777Inside user-oriented325.7190.809Other115.8000.820Total755.7200.793Requirements and standardsSafety-critical106.1001.0781.4320.241Commercial225.8940.772Inside user-oriented325.9171.088Other115.2731.191Total755.8401.029Program utilitySafety-critical105.7500.6771.0570.373Commercial225.8640.915Inside user-oriented325.4530.892Other115.4551.150Total755.6130.917Table 9
Results of ANOVA for Hypothesis2.
New principleDemographic factorNMSDFPTesting abilityDatabase135.4360.8750.4520.716Operation system86.1671.098Communication control115.3941.307Language processor165.6880.715Other275.7041.167Total755.6581.048Programmer abilityDatabase135.3380.6991.0780.374Operation system85.8250.713Communication control115.7821.126Language processor165.9250.669Other275.7260.759Total755.7200.793Requirements and standardsDatabase135.4871.1021.9970.104Operation system86.5420.562Communication control115.7881.167Language processor165.5211.109Other276.0120.908Total755.8401.029Program utilityDatabase135.2690.8814.6390.002∗∗Operation system86.3130.704Communication control115.3180.643Language processor165.1561.136Other275.9630.706Total755.6130.917∗∗p<0.01.Table 10
Results of ANOVA for Hypothesis3.
New principleDemographic factorNMSDFPTesting abilityLess than 5 years295.5861.1331.1890.3106–10 years225.9390.767More than 11 years245.4861.150Total755.6581.048Programmer abilityLess than 5 yearsa295.4550.8053.6440.031∗6–10 yearsab225.7360.803More than 11 yearsb246.0250.676Total755.7200.793Requirements and standardsLess than 5 years295.8970.9720.6270.5376–10 years225.6361.122More than 11 years245.9581.023Total755.8401.029Program utilityLess than 5 years295.5690.9330.1520.8596–10 years225.7051.008More than 11 years245.5830.843Total755.6130.917a,bThe same group of Scheffe’s test; ∗p<0.05.Table 11
Results of ANOVA for Hypothesis4.
New principleDemographic factorNMSDFPTesting abilityManager115.2420.4490.5170.763System engineer105.6331.310Programmer225.7730.825Tester115.5451.537Administrator115.8481.158Other105.8001.033Total755.6581.048Programmer abilityManager115.6000.7210.9910.430System engineer105.6800.598Programmer225.4910.758Tester115.8001.070Administrator116.0550.722Other105.9400.833Total755.7200.793Requirements and standardsManager115.4241.1061.3780.243System engineer105.8670.652Programmer225.6361.163Tester116.4240.668Administrator116.0611.063Other105.8331.103Total755.8401.029Program utilityManager115.0451.1282.6150.032∗System engineer106.1000.615Programmer225.3640.640Tester116.0910.944Administrator115.6821.031Other105.7000.949Total755.6130.917∗p<0.05.Table 12
Results of ANOVA for Hypothesis5.
Demographic factorNMSDFPPercentage of reused codeLess than 5 yearsa2960.69019.07312.662<0.001∗∗∗6–10 yearsb2247.72717.164More than 11 yearsb2436.45815.776Total7549.13320.091a,bThe same group of Scheffe’s test.Figure 1
Results of ANOVA for Hypothesis1.
(a)(b)(c)(d)Figure 2
Results of ANOVA for Hypothesis2.
(a)(b)(c)(d)Figure 3
Results of ANOVA for Hypothesis3.
(a)(b)(c)(d)Figure 4
Results of ANOVA for Hypothesis4.
(a)(b)(c)(d)Figure 5
Results of ANOVA for Hypothesis5.


#### 3.2.3. Regression Analysis

A regression analysis was performed to determine the effect of the four constituent factors obtained through factor analysis on the improvement of the software reliability accuracy. Table13 shows the correlation between the four component factors in the factor analysis. As can be seen in Table 13, testing ability is correlated with requirements, standards, and program utility, whereas programmer ability is correlated with requirements and standards. Furthermore, requirements and standards have a correlation with all and program utility has a correlation with testing ability and requirements and standards. Regression analysis was performed using a stepwise method to examine the influence of the constructs on the improvement of software reliability accuracy. Figure 6 shows the correlation results for the four new component factors.Table 13
Correlation of the four component factors.
New principleTesting abilityProgrammer abilityRequirements and standardsProgram utilityTesting ability10.0450.340∗∗0.394∗∗Programmer ability0.04510.361∗∗0.067Requirements and standards0.340∗∗0.361∗∗10.330∗∗Program utility0.394∗∗0.0670.330∗∗1∗∗p<0.01; ∗∗∗p<0.001.Figure 6
Results of correlation for the four new component factors.Table14 presents the results of the regression analysis. As a rule of thumb, with respect to the p values for our predictors, we say that the b coefficient is statistically significant if its p value is smaller than 0.05. As can be seen in Table 14, two factors of the four component factors were excluded because the p value was less than 0.05. The b coefficients indicate how many unit component factors increase for a single unit increase in each predictor. Hence, a 1-point increase in testing ability corresponds to a 0.058-point increase. The beta coefficients allow us to compare the relative strengths of the predictors.Table 14
Results of regression analysis.
NameUnstandardized coefficientsStandardized coefficientstR2FBStd. errorBeta(Constant)−0.0090.143−0.0620.27513.679∗∗∗Testing ability0.0580.0150.3753.738∗Programmer ability0.0710.0200.3503.487∗∗∗∗p<0.01; ∗∗∗p<0.001.


### 3.3. Analysis of Development Phase

#### 3.3.1. Analysis between Environmental Factor Groups

We want to know how the four software development phases affect each other except the hardware systems phase in the five groups categorized by Zhang and Pham [13]. As a result, there was a significant difference between the four groups, as shown in Table 15. Tukey grouping of the post-hoc analysis was applied to group different development phases based on the mean value of participants’ scores; however, there was no significant difference between each group. The mean score of the analysis and design phase was the highest at 5.545, and the mean score of the coding phase was the lowest at 5.247. The mean scores of the general and testing phases were 5.497 and 5.265, respectively.Table 15
Final grouping for development phase.
PhaseTukey groupingNMSDFPGeneralA755.4970.5883.0860.028∗Analysis and designA755.5450.793CodingA755.2470.739TestingA755.2650.895∗p<0.05.


#### 3.3.2. Significant Factor for Each Development Phase

In Section1, we discuss software development projects that are subject to frequent modifications. As a result, SDEFs hinder communication between users, developers, and testers. Thus, every development phase requires data on significant SDEFs for software reliability assessment. This study applies a stepwise elimination method to eliminate nonsignificant SDEFs in each development phase. The variables of stepwise elimination for each development phase were environmental factors in this phase. Table 16 presents the significant environmental factors, parameter estimates, t-values, p values, and so on, for the significant environmental factors in each phase. The results show that significant environmental factors are positively correlated with software reliability improvement. In the analysis and design phase, f14, development management is one of the significant environmental factors. F18, program workload, and f20, human nature, are significant environmental factors in the coding phase. Moreover, f22, the testing effort, is also a significant environmental factor. However, in the general phase, there were no significant environmental factors. F18, program workload; f20, human nature; and f22, testing effort, have high rankings in Table 3.Table 16
Significant factor for each development phase.
PhaseSignificant EFs (name)Unstandardized coefficientsStandardized coefficientstR2FBStd. errorBetaGeneral———————Analysis and design(Constant)0.4770.0835.7710.1149.349∗∗f14 (development management)0.0450.0150.3373.058Coding(Constant)0.3210.0953.3690.2059.269∗∗∗f18 (program workload)0.0360.0150.2832.497f20 (human nature)0.0350.0150.2632.321Testing(Constant)0.3680.0884.1640.18816.871f22 (testing effort)0.0630.0150.4334.107∗∗p<0.01; ∗∗∗p<0.001.


## 3.1. Relative Importance of SDEFs

### 3.1.1. Importance of SDEFs Based on the Relative Weighted Method

The survey consisted of 32 SDEFs, with 7 points representing the most important factor in software reliability and 0 points indicating the least important. The relative weighting method with each score was applied to determine the ranking [13]. Table 4 shows the results of analyzing the ranking of 32 software development SDEFs using the relative weighting method. Typically, an environmental factor with a higher normalized weight has a more significant impact on the software reliability assessment than the factor that has a lower weight. Therefore, software developers should pay more attention to SDEFs with higher normalized weights. As can be seen in Table 4, in Korean companies, the normalized weight value of factor f11 (requirement analysis) is the highest at 0.03667, followed by f15 (programmer skill) at 0.03634, f12 (relationship of detailed design to requirement) at 0.03578, f20 (human nature) at 0.03449, f22 (testing effort) at 0.03449, and f24 (testing methodologies) at 0.03449.Table 4
The results of normalized weighted ranking.
RankFactorFactor nameNormalized weight1f11Requirements analysis0.036672f15Programmer skill0.036343f12Relationship of detailed design to requirement0.035784f20Human nature0.034495f22Testing effort0.034496f24Testing methodologies0.034497f18Program workload (stress)0.034328f6Percentage of reused modules0.034249f19Domain knowledge0.0340810f7Programming language0.0340011f25Testing coverage0.0338412f2Program categories0.0337613f13Work standards0.0336814f5Level of programming technologies0.0335215f14Development management0.0334416f8Frequency of program specification change0.0333517f1Program complexity0.0328718f4Amount of programming effort0.0328719f21Testing environment0.0322220f3Difficulty of programming0.0318221f9Volume of program design documents0.0312622f10Design methodology0.0309323f17Development team size0.0302124f23Testing resource allocation0.0297225f27Documentation0.0296426f26Testing tools0.0288327f29Storage devices0.0255228f32System software0.0243129f30Input/output devices0.0235830f28Processors0.0222931f31Telecommunication devices0.0222132f16Programmer organization0.02124


### 3.1.2. Correlation Analysis between SDEFs

This section describes a correlation analysis performed to examine the correlation between SDEFs. The Pearson correlation coefficient (Pearson’sr) is a measure of the linear dependence between two variables, where 1 is the total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation [15]. Table 5 shows the correlation of the top-10 SDEFs with high normalized weight values in Table 4. Correlation analysis was performed using 32 SDEFs; only the SDEFs that were significant in the 10 SDEFs shown in Table 5 were tabulated and the other SDEFs were excluded, and only those factors with a strong correlation were summarized (r > 0.4 or r < −0.4). As a fragmentary example, f11 (requirements analysis) and f12 (relationship of detailed design to requirement), and f11 (requirements analysis) and f13 (work standards) have a strong quantitative linear relationship.Table 5
Correlation analysis between SDEFs.
Factor (factor name)Correlated factor (factor name)Pearson’srf11 (requirements analysis)f12 (relationship of detailed design to requirement)0.656∗∗f13 (work standards)0.598∗∗f10 (design methodology)0.541∗∗f17 (development team size)0.491∗∗f14 (development management)0.464∗∗f18 (program workload (stress))0.451∗∗f6 (percentage of reused modules)0.421∗∗f9 (volume of program design documents)0.412∗∗f25 (testing coverage)0.400∗∗f15 (programmer skill)f19 (domain knowledge)0.629∗∗f3 (difficulty of programming)0.533∗∗f20 (human nature)0.423∗∗f12 (relationship of detailed design to requirement)f11 (requirements analysis)0.656∗∗f10 (design methodology)0.406∗∗f20 (human nature)f7 (programming language)0.453∗∗f14 (development management)0.439∗∗f15 (programmer skill)0.423∗∗f16 (programmer organization)0.407∗∗f22 (testing effort)f25 (testing coverage)0.741∗∗f24 (testing methodologies)0.728∗∗f27 (documentation)0.438∗∗f28 (processors)−0.419∗∗f25 (testing methodologies)f25 (testing coverage)0.843∗∗F22 (testing effort)0.728∗∗f27 (documentation)0.659∗∗F26 (testing tools)0.503∗∗f23 (testing resource allocation)0.420∗∗f28 (processors)−0.419∗∗f18 (program workload (stress))f14 (development management)0.453∗∗f11 (requirements analysis)0.451∗∗f5 (level of programming technologies)0.449∗∗f13 (work standards)0.414∗∗f6 (percentage of reused modules)f11 (requirements analysis)0.421∗∗f10 (design methodology)0.414∗∗f19 (domain knowledge)F15 (programmer skill)0.629∗∗f3 (difficulty of programming)0.426∗∗f5 (level of programming technologies)0.426∗∗f7 (programming language)f20 (human nature)0.453∗∗f14 (development management)0.411∗∗f17 (development team size)0.403∗∗∗∗p<0.01.


## 3.1.1. Importance of SDEFs Based on the Relative Weighted Method

The survey consisted of 32 SDEFs, with 7 points representing the most important factor in software reliability and 0 points indicating the least important. The relative weighting method with each score was applied to determine the ranking [13]. Table 4 shows the results of analyzing the ranking of 32 software development SDEFs using the relative weighting method. Typically, an environmental factor with a higher normalized weight has a more significant impact on the software reliability assessment than the factor that has a lower weight. Therefore, software developers should pay more attention to SDEFs with higher normalized weights. As can be seen in Table 4, in Korean companies, the normalized weight value of factor f11 (requirement analysis) is the highest at 0.03667, followed by f15 (programmer skill) at 0.03634, f12 (relationship of detailed design to requirement) at 0.03578, f20 (human nature) at 0.03449, f22 (testing effort) at 0.03449, and f24 (testing methodologies) at 0.03449.Table 4
The results of normalized weighted ranking.
RankFactorFactor nameNormalized weight1f11Requirements analysis0.036672f15Programmer skill0.036343f12Relationship of detailed design to requirement0.035784f20Human nature0.034495f22Testing effort0.034496f24Testing methodologies0.034497f18Program workload (stress)0.034328f6Percentage of reused modules0.034249f19Domain knowledge0.0340810f7Programming language0.0340011f25Testing coverage0.0338412f2Program categories0.0337613f13Work standards0.0336814f5Level of programming technologies0.0335215f14Development management0.0334416f8Frequency of program specification change0.0333517f1Program complexity0.0328718f4Amount of programming effort0.0328719f21Testing environment0.0322220f3Difficulty of programming0.0318221f9Volume of program design documents0.0312622f10Design methodology0.0309323f17Development team size0.0302124f23Testing resource allocation0.0297225f27Documentation0.0296426f26Testing tools0.0288327f29Storage devices0.0255228f32System software0.0243129f30Input/output devices0.0235830f28Processors0.0222931f31Telecommunication devices0.0222132f16Programmer organization0.02124


## 3.1.2. Correlation Analysis between SDEFs

This section describes a correlation analysis performed to examine the correlation between SDEFs. The Pearson correlation coefficient (Pearson’sr) is a measure of the linear dependence between two variables, where 1 is the total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation [15]. Table 5 shows the correlation of the top-10 SDEFs with high normalized weight values in Table 4. Correlation analysis was performed using 32 SDEFs; only the SDEFs that were significant in the 10 SDEFs shown in Table 5 were tabulated and the other SDEFs were excluded, and only those factors with a strong correlation were summarized (r > 0.4 or r < −0.4). As a fragmentary example, f11 (requirements analysis) and f12 (relationship of detailed design to requirement), and f11 (requirements analysis) and f13 (work standards) have a strong quantitative linear relationship.Table 5
Correlation analysis between SDEFs.
Factor (factor name)Correlated factor (factor name)Pearson’srf11 (requirements analysis)f12 (relationship of detailed design to requirement)0.656∗∗f13 (work standards)0.598∗∗f10 (design methodology)0.541∗∗f17 (development team size)0.491∗∗f14 (development management)0.464∗∗f18 (program workload (stress))0.451∗∗f6 (percentage of reused modules)0.421∗∗f9 (volume of program design documents)0.412∗∗f25 (testing coverage)0.400∗∗f15 (programmer skill)f19 (domain knowledge)0.629∗∗f3 (difficulty of programming)0.533∗∗f20 (human nature)0.423∗∗f12 (relationship of detailed design to requirement)f11 (requirements analysis)0.656∗∗f10 (design methodology)0.406∗∗f20 (human nature)f7 (programming language)0.453∗∗f14 (development management)0.439∗∗f15 (programmer skill)0.423∗∗f16 (programmer organization)0.407∗∗f22 (testing effort)f25 (testing coverage)0.741∗∗f24 (testing methodologies)0.728∗∗f27 (documentation)0.438∗∗f28 (processors)−0.419∗∗f25 (testing methodologies)f25 (testing coverage)0.843∗∗F22 (testing effort)0.728∗∗f27 (documentation)0.659∗∗F26 (testing tools)0.503∗∗f23 (testing resource allocation)0.420∗∗f28 (processors)−0.419∗∗f18 (program workload (stress))f14 (development management)0.453∗∗f11 (requirements analysis)0.451∗∗f5 (level of programming technologies)0.449∗∗f13 (work standards)0.414∗∗f6 (percentage of reused modules)f11 (requirements analysis)0.421∗∗f10 (design methodology)0.414∗∗f19 (domain knowledge)F15 (programmer skill)0.629∗∗f3 (difficulty of programming)0.426∗∗f5 (level of programming technologies)0.426∗∗f7 (programming language)f20 (human nature)0.453∗∗f14 (development management)0.411∗∗f17 (development team size)0.403∗∗∗∗p<0.01.


## 3.2. Environmental Factor Analysis

### 3.2.1. Factor Analysis

Factor analysis was conducted to verify the construct validity of the questionnaire. Construct validity is determined by the suitability of the items included in the questionnaire to evaluate the set of hypothesized constructs. Factorial validity is a construct validity that proves the validity through factor analysis. Therefore, factor analysis was conducted to verify the validity of the items related to the improvement of the software reliability assessment [16]. Factor analysis was performed using the principal component method, and the varimax method was used as the orthogonal rotation method. The eigenvalues of the components, proportion of variance, and cumulative proportion of variance are presented in Table 6, and 70.973% of the variation can be explained by the first four components. The first to fourth eigenvalues are greater than 1; however, the value after the fifth eigenvalue is less than 1. Those with eigenvalues less than 1 are not considered to be stable. They account for less variability than a single variable does, and hence, they are not retained in the analysis. In this sense, we end up with fewer factors than the original number of variables [17]; hence, the first four components are retained. As a result, the selected factors are equal to the top 13 most important SDEFs based on the ranking results presented by the relative weighting method in Table 4. We can observe that the SDEFs cover many aspects of the development and testing of software, and the majority of these factors have a similarly significant impact on software reliability.Table 6
Eigenvalue of the correlation matrix.
ComponentsEigenvalueProportion of varianceCumulative proportion14.04131.08231.08222.68220.63151.71331.39310.71762.43041.1118.54370.97350.8356.42077.39360.7305.61883.01170.5063.89686.90780.4993.84190.74790.3592.75893.505100.3002.31095.814110.2652.04197.856120.1491.14799.003130.1300.997100.000As shown in Table7, we retained four principal components: PCA1, PCA2, PCA3, and PCA4. The first component, PCA1, consisted of factors for f24, f25, and f22, which are related to testing ability, whereas PCA2 consists of factors for f15, f19, f20, f7, and f18, which are related to programmer ability. PCA3 consisted of factors for f11, f12, and f13, whereas PCA4 consisted of factors f2 and f6, which were related to requirements, standards, and program utility, respectively. All four components had high loading values, which exceeded 0.5. Several methods have been proposed as a measure of reliability, but in this study, Cronbach’s α test was used to test the internal consistency of the items. Cronbach’s α typically increases as the intercorrelations among test items increase and is, thus, known as an internal consistency estimate of the reliability of test scores. If the Cronbach’s α coefficient is more than 0.6, the scale is reliable [18]. As can be seen in Table 7, Cronbach’s α is 0.786, which is greater than 0.6.Table 7
New principles for each principal component.
ComponentNew principleFactor (name)LoadingRotation sums of squared loadingsCronbach’sαEigenvalueProportion of varianceCumulative proportionPCA1Testing abilityf24 (testing methodologies)0.8982.76421.26221.2620.786f25 (testing coverage)0.895f22 (testing effort)0.857PCA2Programmer abilityf15 (programmer skill)0.8052.49919.22540.486f19 (domain knowledge)0.731f20 (human nature)0.718f7 (programming language)0.618f18 (program workload)0.531PCA3Requirements and standardsf11 (requirements analysis)0.8712.20216.94157.427f12 (relationship of detailed design to requirement)0.827f13 (work standards)0.642PCA4Program utilityf2 (program categories)0.7491.76113.54570.973f6 (percentage of reused modules)0.682


### 3.2.2. Hypothesis Test

A statistical hypothesis is a method of statistical interference that is testable based on observing a process that is modeled via a set of random variables [19]. Hypothesis tests are used to determine the outcomes of a study that would lead to a rejection of the null hypothesis for a pre-specified level of significance. The process of distinguishing between the null hypothesis and the alternative hypothesis is aided by identifying two conceptual types of errors, such as type 1 and type 2, and by specifying parametric limits on, for instance, how much type 1 error will be permitted. The null hypothesis H0 is a simple hypothesis related to the contradiction, with the theory to be proved. The alternative hypothesis H1 is a hypothesis related to the theory to be proved. The hypothesis will reveal whether their opinion can be considered “the same.”Hypothesis 1.
People developing software for different applications have the same opinion on the importance of the four components of the new principle. Safety-critical, commercial, inside-used systems, and others may not have the same opinion on the significance of all these factors.(1)H0:μs=μc=μi=μo,H1:atleast onμiis different.Hypothesis 2.
People with different types of software development experience have the same opinion on the significance of the four components of the new principle. The database, operation system, communication control, language processor, and others are considered.(2)H0:μd=μo=μc=μl=μo,H1:atleast onμiis different.Hypothesis 3.
People playing different numbers of years of software development have the same opinion on the significance of the four components of the new principle. Less than 5 years, 6–10 years, and more than 11 years are considered here.(3)H0:μ≤5=μ6−10=μ≥11,H1:atleast onμiis different.Hypothesis 4.
People playing different roles in software development have the same opinion on the significance of the four components of the new principle. Managers, system engineers, programmers, testers, administrators, and others are considered here.(4)H0:μm=μs=μp=μt=μa=μo,H1:atleast onμiis different.Hypothesis 5.
People playing different numbers of years of software development have the same opinion on the significance of the percentage of reused code. Less than 5 years, 6–10 years, and more than 11 years are considered here.(5)H0:μ≤5=μ6−10=μ≥11,H1:atleast onμiis different.
We performed ANOVA on hypotheses 1 to 5. ANOVAs are useful for comparing three or more means (groups or variables) for statistical significance. ANOVA is suitable for a wide range of practical problems. The results of the analysis for the given hypotheses are presented in Tables8–12. As per the results of Hypothesis 1, there was no significant difference according to the group, as shown in Table 8. As per the results of Hypothesis 2, there is a significant difference only in the group of program utility component factors, as shown in Table 9. A post-hoc analysis was performed using Scheffe’s test method; however, there was no significant difference between the groups. As per the results of Hypothesis 3, there is a significant difference only in the programmer ability component factor group, as shown in Table 10. In the post-hoc analysis, there was a significant difference between groups of less than 5 years and more than 11 years. As per the results of Hypothesis 4, there is a significant difference only in the group of program utility component factors, as shown in Table 11. Post-hoc analysis showed no significant differences between the groups. According to the results of Hypothesis 5, there is a significant difference in the percentage of reused code, as shown in Table 12. In the post-hoc analysis, there was a significant difference between groups of less than 5 years and 5–10 years, and less than 5 years and more than 11 years. In Tables 8–12, N is the frequency, M is the mean value, SD is the standard deviation, F is the variance between groups indicating the degree of difference between groups, and P is the p value. Figures 1–5 show the results of ANOVA for Hypothesis 1–5.Table 8
Results of ANOVA for Hypothesis1.
New principleDemographic factorNMSDFPTesting abilitySafety-critical105.8331.0330.4520.716Commercial225.8181.246Inside user-oriented325.5311.040Other115.5450.637Total755.6581.048Programmer abilitySafety-critical105.4800.8180.3920.759Commercial225.7910.777Inside user-oriented325.7190.809Other115.8000.820Total755.7200.793Requirements and standardsSafety-critical106.1001.0781.4320.241Commercial225.8940.772Inside user-oriented325.9171.088Other115.2731.191Total755.8401.029Program utilitySafety-critical105.7500.6771.0570.373Commercial225.8640.915Inside user-oriented325.4530.892Other115.4551.150Total755.6130.917Table 9
Results of ANOVA for Hypothesis2.
New principleDemographic factorNMSDFPTesting abilityDatabase135.4360.8750.4520.716Operation system86.1671.098Communication control115.3941.307Language processor165.6880.715Other275.7041.167Total755.6581.048Programmer abilityDatabase135.3380.6991.0780.374Operation system85.8250.713Communication control115.7821.126Language processor165.9250.669Other275.7260.759Total755.7200.793Requirements and standardsDatabase135.4871.1021.9970.104Operation system86.5420.562Communication control115.7881.167Language processor165.5211.109Other276.0120.908Total755.8401.029Program utilityDatabase135.2690.8814.6390.002∗∗Operation system86.3130.704Communication control115.3180.643Language processor165.1561.136Other275.9630.706Total755.6130.917∗∗p<0.01.Table 10
Results of ANOVA for Hypothesis3.
New principleDemographic factorNMSDFPTesting abilityLess than 5 years295.5861.1331.1890.3106–10 years225.9390.767More than 11 years245.4861.150Total755.6581.048Programmer abilityLess than 5 yearsa295.4550.8053.6440.031∗6–10 yearsab225.7360.803More than 11 yearsb246.0250.676Total755.7200.793Requirements and standardsLess than 5 years295.8970.9720.6270.5376–10 years225.6361.122More than 11 years245.9581.023Total755.8401.029Program utilityLess than 5 years295.5690.9330.1520.8596–10 years225.7051.008More than 11 years245.5830.843Total755.6130.917a,bThe same group of Scheffe’s test; ∗p<0.05.Table 11
Results of ANOVA for Hypothesis4.
New principleDemographic factorNMSDFPTesting abilityManager115.2420.4490.5170.763System engineer105.6331.310Programmer225.7730.825Tester115.5451.537Administrator115.8481.158Other105.8001.033Total755.6581.048Programmer abilityManager115.6000.7210.9910.430System engineer105.6800.598Programmer225.4910.758Tester115.8001.070Administrator116.0550.722Other105.9400.833Total755.7200.793Requirements and standardsManager115.4241.1061.3780.243System engineer105.8670.652Programmer225.6361.163Tester116.4240.668Administrator116.0611.063Other105.8331.103Total755.8401.029Program utilityManager115.0451.1282.6150.032∗System engineer106.1000.615Programmer225.3640.640Tester116.0910.944Administrator115.6821.031Other105.7000.949Total755.6130.917∗p<0.05.Table 12
Results of ANOVA for Hypothesis5.
Demographic factorNMSDFPPercentage of reused codeLess than 5 yearsa2960.69019.07312.662<0.001∗∗∗6–10 yearsb2247.72717.164More than 11 yearsb2436.45815.776Total7549.13320.091a,bThe same group of Scheffe’s test.Figure 1
Results of ANOVA for Hypothesis1.
(a)(b)(c)(d)Figure 2
Results of ANOVA for Hypothesis2.
(a)(b)(c)(d)Figure 3
Results of ANOVA for Hypothesis3.
(a)(b)(c)(d)Figure 4
Results of ANOVA for Hypothesis4.
(a)(b)(c)(d)Figure 5
Results of ANOVA for Hypothesis5.


### 3.2.3. Regression Analysis

A regression analysis was performed to determine the effect of the four constituent factors obtained through factor analysis on the improvement of the software reliability accuracy. Table13 shows the correlation between the four component factors in the factor analysis. As can be seen in Table 13, testing ability is correlated with requirements, standards, and program utility, whereas programmer ability is correlated with requirements and standards. Furthermore, requirements and standards have a correlation with all and program utility has a correlation with testing ability and requirements and standards. Regression analysis was performed using a stepwise method to examine the influence of the constructs on the improvement of software reliability accuracy. Figure 6 shows the correlation results for the four new component factors.Table 13
Correlation of the four component factors.
New principleTesting abilityProgrammer abilityRequirements and standardsProgram utilityTesting ability10.0450.340∗∗0.394∗∗Programmer ability0.04510.361∗∗0.067Requirements and standards0.340∗∗0.361∗∗10.330∗∗Program utility0.394∗∗0.0670.330∗∗1∗∗p<0.01; ∗∗∗p<0.001.Figure 6
Results of correlation for the four new component factors.Table14 presents the results of the regression analysis. As a rule of thumb, with respect to the p values for our predictors, we say that the b coefficient is statistically significant if its p value is smaller than 0.05. As can be seen in Table 14, two factors of the four component factors were excluded because the p value was less than 0.05. The b coefficients indicate how many unit component factors increase for a single unit increase in each predictor. Hence, a 1-point increase in testing ability corresponds to a 0.058-point increase. The beta coefficients allow us to compare the relative strengths of the predictors.Table 14
Results of regression analysis.
NameUnstandardized coefficientsStandardized coefficientstR2FBStd. errorBeta(Constant)−0.0090.143−0.0620.27513.679∗∗∗Testing ability0.0580.0150.3753.738∗Programmer ability0.0710.0200.3503.487∗∗∗∗p<0.01; ∗∗∗p<0.001.


## 3.2.1. Factor Analysis

Factor analysis was conducted to verify the construct validity of the questionnaire. Construct validity is determined by the suitability of the items included in the questionnaire to evaluate the set of hypothesized constructs. Factorial validity is a construct validity that proves the validity through factor analysis. Therefore, factor analysis was conducted to verify the validity of the items related to the improvement of the software reliability assessment [16]. Factor analysis was performed using the principal component method, and the varimax method was used as the orthogonal rotation method. The eigenvalues of the components, proportion of variance, and cumulative proportion of variance are presented in Table 6, and 70.973% of the variation can be explained by the first four components. The first to fourth eigenvalues are greater than 1; however, the value after the fifth eigenvalue is less than 1. Those with eigenvalues less than 1 are not considered to be stable. They account for less variability than a single variable does, and hence, they are not retained in the analysis. In this sense, we end up with fewer factors than the original number of variables [17]; hence, the first four components are retained. As a result, the selected factors are equal to the top 13 most important SDEFs based on the ranking results presented by the relative weighting method in Table 4. We can observe that the SDEFs cover many aspects of the development and testing of software, and the majority of these factors have a similarly significant impact on software reliability.Table 6
Eigenvalue of the correlation matrix.
ComponentsEigenvalueProportion of varianceCumulative proportion14.04131.08231.08222.68220.63151.71331.39310.71762.43041.1118.54370.97350.8356.42077.39360.7305.61883.01170.5063.89686.90780.4993.84190.74790.3592.75893.505100.3002.31095.814110.2652.04197.856120.1491.14799.003130.1300.997100.000As shown in Table7, we retained four principal components: PCA1, PCA2, PCA3, and PCA4. The first component, PCA1, consisted of factors for f24, f25, and f22, which are related to testing ability, whereas PCA2 consists of factors for f15, f19, f20, f7, and f18, which are related to programmer ability. PCA3 consisted of factors for f11, f12, and f13, whereas PCA4 consisted of factors f2 and f6, which were related to requirements, standards, and program utility, respectively. All four components had high loading values, which exceeded 0.5. Several methods have been proposed as a measure of reliability, but in this study, Cronbach’s α test was used to test the internal consistency of the items. Cronbach’s α typically increases as the intercorrelations among test items increase and is, thus, known as an internal consistency estimate of the reliability of test scores. If the Cronbach’s α coefficient is more than 0.6, the scale is reliable [18]. As can be seen in Table 7, Cronbach’s α is 0.786, which is greater than 0.6.Table 7
New principles for each principal component.
ComponentNew principleFactor (name)LoadingRotation sums of squared loadingsCronbach’sαEigenvalueProportion of varianceCumulative proportionPCA1Testing abilityf24 (testing methodologies)0.8982.76421.26221.2620.786f25 (testing coverage)0.895f22 (testing effort)0.857PCA2Programmer abilityf15 (programmer skill)0.8052.49919.22540.486f19 (domain knowledge)0.731f20 (human nature)0.718f7 (programming language)0.618f18 (program workload)0.531PCA3Requirements and standardsf11 (requirements analysis)0.8712.20216.94157.427f12 (relationship of detailed design to requirement)0.827f13 (work standards)0.642PCA4Program utilityf2 (program categories)0.7491.76113.54570.973f6 (percentage of reused modules)0.682


## 3.2.2. Hypothesis Test

A statistical hypothesis is a method of statistical interference that is testable based on observing a process that is modeled via a set of random variables [19]. Hypothesis tests are used to determine the outcomes of a study that would lead to a rejection of the null hypothesis for a pre-specified level of significance. The process of distinguishing between the null hypothesis and the alternative hypothesis is aided by identifying two conceptual types of errors, such as type 1 and type 2, and by specifying parametric limits on, for instance, how much type 1 error will be permitted. The null hypothesis H0 is a simple hypothesis related to the contradiction, with the theory to be proved. The alternative hypothesis H1 is a hypothesis related to the theory to be proved. The hypothesis will reveal whether their opinion can be considered “the same.”Hypothesis 1.
People developing software for different applications have the same opinion on the importance of the four components of the new principle. Safety-critical, commercial, inside-used systems, and others may not have the same opinion on the significance of all these factors.(1)H0:μs=μc=μi=μo,H1:atleast onμiis different.Hypothesis 2.
People with different types of software development experience have the same opinion on the significance of the four components of the new principle. The database, operation system, communication control, language processor, and others are considered.(2)H0:μd=μo=μc=μl=μo,H1:atleast onμiis different.Hypothesis 3.
People playing different numbers of years of software development have the same opinion on the significance of the four components of the new principle. Less than 5 years, 6–10 years, and more than 11 years are considered here.(3)H0:μ≤5=μ6−10=μ≥11,H1:atleast onμiis different.Hypothesis 4.
People playing different roles in software development have the same opinion on the significance of the four components of the new principle. Managers, system engineers, programmers, testers, administrators, and others are considered here.(4)H0:μm=μs=μp=μt=μa=μo,H1:atleast onμiis different.Hypothesis 5.
People playing different numbers of years of software development have the same opinion on the significance of the percentage of reused code. Less than 5 years, 6–10 years, and more than 11 years are considered here.(5)H0:μ≤5=μ6−10=μ≥11,H1:atleast onμiis different.
We performed ANOVA on hypotheses 1 to 5. ANOVAs are useful for comparing three or more means (groups or variables) for statistical significance. ANOVA is suitable for a wide range of practical problems. The results of the analysis for the given hypotheses are presented in Tables8–12. As per the results of Hypothesis 1, there was no significant difference according to the group, as shown in Table 8. As per the results of Hypothesis 2, there is a significant difference only in the group of program utility component factors, as shown in Table 9. A post-hoc analysis was performed using Scheffe’s test method; however, there was no significant difference between the groups. As per the results of Hypothesis 3, there is a significant difference only in the programmer ability component factor group, as shown in Table 10. In the post-hoc analysis, there was a significant difference between groups of less than 5 years and more than 11 years. As per the results of Hypothesis 4, there is a significant difference only in the group of program utility component factors, as shown in Table 11. Post-hoc analysis showed no significant differences between the groups. According to the results of Hypothesis 5, there is a significant difference in the percentage of reused code, as shown in Table 12. In the post-hoc analysis, there was a significant difference between groups of less than 5 years and 5–10 years, and less than 5 years and more than 11 years. In Tables 8–12, N is the frequency, M is the mean value, SD is the standard deviation, F is the variance between groups indicating the degree of difference between groups, and P is the p value. Figures 1–5 show the results of ANOVA for Hypothesis 1–5.Table 8
Results of ANOVA for Hypothesis1.
New principleDemographic factorNMSDFPTesting abilitySafety-critical105.8331.0330.4520.716Commercial225.8181.246Inside user-oriented325.5311.040Other115.5450.637Total755.6581.048Programmer abilitySafety-critical105.4800.8180.3920.759Commercial225.7910.777Inside user-oriented325.7190.809Other115.8000.820Total755.7200.793Requirements and standardsSafety-critical106.1001.0781.4320.241Commercial225.8940.772Inside user-oriented325.9171.088Other115.2731.191Total755.8401.029Program utilitySafety-critical105.7500.6771.0570.373Commercial225.8640.915Inside user-oriented325.4530.892Other115.4551.150Total755.6130.917Table 9
Results of ANOVA for Hypothesis2.
New principleDemographic factorNMSDFPTesting abilityDatabase135.4360.8750.4520.716Operation system86.1671.098Communication control115.3941.307Language processor165.6880.715Other275.7041.167Total755.6581.048Programmer abilityDatabase135.3380.6991.0780.374Operation system85.8250.713Communication control115.7821.126Language processor165.9250.669Other275.7260.759Total755.7200.793Requirements and standardsDatabase135.4871.1021.9970.104Operation system86.5420.562Communication control115.7881.167Language processor165.5211.109Other276.0120.908Total755.8401.029Program utilityDatabase135.2690.8814.6390.002∗∗Operation system86.3130.704Communication control115.3180.643Language processor165.1561.136Other275.9630.706Total755.6130.917∗∗p<0.01.Table 10
Results of ANOVA for Hypothesis3.
New principleDemographic factorNMSDFPTesting abilityLess than 5 years295.5861.1331.1890.3106–10 years225.9390.767More than 11 years245.4861.150Total755.6581.048Programmer abilityLess than 5 yearsa295.4550.8053.6440.031∗6–10 yearsab225.7360.803More than 11 yearsb246.0250.676Total755.7200.793Requirements and standardsLess than 5 years295.8970.9720.6270.5376–10 years225.6361.122More than 11 years245.9581.023Total755.8401.029Program utilityLess than 5 years295.5690.9330.1520.8596–10 years225.7051.008More than 11 years245.5830.843Total755.6130.917a,bThe same group of Scheffe’s test; ∗p<0.05.Table 11
Results of ANOVA for Hypothesis4.
New principleDemographic factorNMSDFPTesting abilityManager115.2420.4490.5170.763System engineer105.6331.310Programmer225.7730.825Tester115.5451.537Administrator115.8481.158Other105.8001.033Total755.6581.048Programmer abilityManager115.6000.7210.9910.430System engineer105.6800.598Programmer225.4910.758Tester115.8001.070Administrator116.0550.722Other105.9400.833Total755.7200.793Requirements and standardsManager115.4241.1061.3780.243System engineer105.8670.652Programmer225.6361.163Tester116.4240.668Administrator116.0611.063Other105.8331.103Total755.8401.029Program utilityManager115.0451.1282.6150.032∗System engineer106.1000.615Programmer225.3640.640Tester116.0910.944Administrator115.6821.031Other105.7000.949Total755.6130.917∗p<0.05.Table 12
Results of ANOVA for Hypothesis5.
Demographic factorNMSDFPPercentage of reused codeLess than 5 yearsa2960.69019.07312.662<0.001∗∗∗6–10 yearsb2247.72717.164More than 11 yearsb2436.45815.776Total7549.13320.091a,bThe same group of Scheffe’s test.Figure 1
Results of ANOVA for Hypothesis1.
(a)(b)(c)(d)Figure 2
Results of ANOVA for Hypothesis2.
(a)(b)(c)(d)Figure 3
Results of ANOVA for Hypothesis3.
(a)(b)(c)(d)Figure 4
Results of ANOVA for Hypothesis4.
(a)(b)(c)(d)Figure 5
Results of ANOVA for Hypothesis5.


## 3.2.3. Regression Analysis

A regression analysis was performed to determine the effect of the four constituent factors obtained through factor analysis on the improvement of the software reliability accuracy. Table13 shows the correlation between the four component factors in the factor analysis. As can be seen in Table 13, testing ability is correlated with requirements, standards, and program utility, whereas programmer ability is correlated with requirements and standards. Furthermore, requirements and standards have a correlation with all and program utility has a correlation with testing ability and requirements and standards. Regression analysis was performed using a stepwise method to examine the influence of the constructs on the improvement of software reliability accuracy. Figure 6 shows the correlation results for the four new component factors.Table 13
Correlation of the four component factors.
New principleTesting abilityProgrammer abilityRequirements and standardsProgram utilityTesting ability10.0450.340∗∗0.394∗∗Programmer ability0.04510.361∗∗0.067Requirements and standards0.340∗∗0.361∗∗10.330∗∗Program utility0.394∗∗0.0670.330∗∗1∗∗p<0.01; ∗∗∗p<0.001.Figure 6
Results of correlation for the four new component factors.Table14 presents the results of the regression analysis. As a rule of thumb, with respect to the p values for our predictors, we say that the b coefficient is statistically significant if its p value is smaller than 0.05. As can be seen in Table 14, two factors of the four component factors were excluded because the p value was less than 0.05. The b coefficients indicate how many unit component factors increase for a single unit increase in each predictor. Hence, a 1-point increase in testing ability corresponds to a 0.058-point increase. The beta coefficients allow us to compare the relative strengths of the predictors.Table 14
Results of regression analysis.
NameUnstandardized coefficientsStandardized coefficientstR2FBStd. errorBeta(Constant)−0.0090.143−0.0620.27513.679∗∗∗Testing ability0.0580.0150.3753.738∗Programmer ability0.0710.0200.3503.487∗∗∗∗p<0.01; ∗∗∗p<0.001.


## 3.3. Analysis of Development Phase

### 3.3.1. Analysis between Environmental Factor Groups

We want to know how the four software development phases affect each other except the hardware systems phase in the five groups categorized by Zhang and Pham [13]. As a result, there was a significant difference between the four groups, as shown in Table 15. Tukey grouping of the post-hoc analysis was applied to group different development phases based on the mean value of participants’ scores; however, there was no significant difference between each group. The mean score of the analysis and design phase was the highest at 5.545, and the mean score of the coding phase was the lowest at 5.247. The mean scores of the general and testing phases were 5.497 and 5.265, respectively.Table 15
Final grouping for development phase.
PhaseTukey groupingNMSDFPGeneralA755.4970.5883.0860.028∗Analysis and designA755.5450.793CodingA755.2470.739TestingA755.2650.895∗p<0.05.


### 3.3.2. Significant Factor for Each Development Phase

In Section1, we discuss software development projects that are subject to frequent modifications. As a result, SDEFs hinder communication between users, developers, and testers. Thus, every development phase requires data on significant SDEFs for software reliability assessment. This study applies a stepwise elimination method to eliminate nonsignificant SDEFs in each development phase. The variables of stepwise elimination for each development phase were environmental factors in this phase. Table 16 presents the significant environmental factors, parameter estimates, t-values, p values, and so on, for the significant environmental factors in each phase. The results show that significant environmental factors are positively correlated with software reliability improvement. In the analysis and design phase, f14, development management is one of the significant environmental factors. F18, program workload, and f20, human nature, are significant environmental factors in the coding phase. Moreover, f22, the testing effort, is also a significant environmental factor. However, in the general phase, there were no significant environmental factors. F18, program workload; f20, human nature; and f22, testing effort, have high rankings in Table 3.Table 16
Significant factor for each development phase.
PhaseSignificant EFs (name)Unstandardized coefficientsStandardized coefficientstR2FBStd. errorBetaGeneral———————Analysis and design(Constant)0.4770.0835.7710.1149.349∗∗f14 (development management)0.0450.0150.3373.058Coding(Constant)0.3210.0953.3690.2059.269∗∗∗f18 (program workload)0.0360.0150.2832.497f20 (human nature)0.0350.0150.2632.321Testing(Constant)0.3680.0884.1640.18816.871f22 (testing effort)0.0630.0150.4334.107∗∗p<0.01; ∗∗∗p<0.001.


## 3.3.1. Analysis between Environmental Factor Groups

We want to know how the four software development phases affect each other except the hardware systems phase in the five groups categorized by Zhang and Pham [13]. As a result, there was a significant difference between the four groups, as shown in Table 15. Tukey grouping of the post-hoc analysis was applied to group different development phases based on the mean value of participants’ scores; however, there was no significant difference between each group. The mean score of the analysis and design phase was the highest at 5.545, and the mean score of the coding phase was the lowest at 5.247. The mean scores of the general and testing phases were 5.497 and 5.265, respectively.Table 15
Final grouping for development phase.
PhaseTukey groupingNMSDFPGeneralA755.4970.5883.0860.028∗Analysis and designA755.5450.793CodingA755.2470.739TestingA755.2650.895∗p<0.05.


## 3.3.2. Significant Factor for Each Development Phase

In Section1, we discuss software development projects that are subject to frequent modifications. As a result, SDEFs hinder communication between users, developers, and testers. Thus, every development phase requires data on significant SDEFs for software reliability assessment. This study applies a stepwise elimination method to eliminate nonsignificant SDEFs in each development phase. The variables of stepwise elimination for each development phase were environmental factors in this phase. Table 16 presents the significant environmental factors, parameter estimates, t-values, p values, and so on, for the significant environmental factors in each phase. The results show that significant environmental factors are positively correlated with software reliability improvement. In the analysis and design phase, f14, development management is one of the significant environmental factors. F18, program workload, and f20, human nature, are significant environmental factors in the coding phase. Moreover, f22, the testing effort, is also a significant environmental factor. However, in the general phase, there were no significant environmental factors. F18, program workload; f20, human nature; and f22, testing effort, have high rankings in Table 3.Table 16
Significant factor for each development phase.
PhaseSignificant EFs (name)Unstandardized coefficientsStandardized coefficientstR2FBStd. errorBetaGeneral———————Analysis and design(Constant)0.4770.0835.7710.1149.349∗∗f14 (development management)0.0450.0150.3373.058Coding(Constant)0.3210.0953.3690.2059.269∗∗∗f18 (program workload)0.0360.0150.2832.497f20 (human nature)0.0350.0150.2632.321Testing(Constant)0.3680.0884.1640.18816.871f22 (testing effort)0.0630.0150.4334.107∗∗p<0.01; ∗∗∗p<0.001.


## 4. Comparison

Zhang and Pham [13] conducted a survey in the early 2000 to assess the importance of SDEFs. Recently, Zhu et al. [20] investigated the changes in the impact of SDEFs. This section compares the results of this study with those of previous studies. By comparing with previous studies, we want to provide software developers working in different countries an important environmental factor in software development to increase the efficiency of work (with respect to work versus time investment) on software reliability improvement.


### 4.1. Comparison of SDEFs Ranking

Since the study uses different population samples from different countries in its three analyses, and the time of the studies is also different, the ranking results differ. It is interesting to see how the ranking of SDEFs according to the relative weighting method changes with time and how it varies according to country. The three results for the top 10 ranks are listed in Table17. As shown in Table 17, the results of Zhang and Pham [13] and Zhu et al. [20] are similar. This can be studied in detail by Zhu et al. [20]. The results of the 10 new ranks also had similar SDEFs. However, there are three different environmental factors: f20, human nature; f18, program workload; and f7 programming language. First, the working area, working time, vacation, holiday, salary, relationship with boss, and culture are different. In Korea, software engineers belong to several categories. The proportion of engineers who occupy the total economy is a part of the medium/large core product development departments, large IT companies, agencies, enterprises, national laboratories, online/mobile game developers, high-growth start-ups, and others. In the middle/large core product development departments, there are numerous high-class talents, and the working time is long, but the treatment is good. The “IT department” of a company is not a core department but rather a department that is responsible for the computerization of a company. Moreover, since agencies begin with price competition, rather than quality from the outset, it is not easy for engineers to receive good treatment. In the case of enterprises and national laboratories, the quality of human resources is high, but the intensity of work is not. However, remuneration is low. In the case of online/mobile game developers, experienced workers receive good treatment, but the average remuneration is not high, as the percentage of junior (1–3 years after graduation) engineers is high. In the case of “high-growth start-ups” in Korea where competition for securing high-quality talent is fierce, treatment is good—this means that developers are working hard to work in a pleasant environment. In comparison to the US Silicon Valley, the proportion of large IT companies, agencies, and online/mobile game developers is relatively low in Korea. IT/infrastructure management is outsourced to low-cost markets, such as India or China, or resorts to cloud services. On the other hand, high value-added industries, which correspond to the medium/large core product development departments, and high-growth start-ups of large enterprises are adequately developing. F11, requirements analysis, is the most important environmental factor in the software development process. The customer’s knowledge level has increased to the extent that the program specifications and other factors vary according to customer requirements. In Korea, consumer requirements are changing faster than in the past. In addition, Korea has superior skills in IT-related industries compared to other countries.Table 17
Comparison of the ranking of SDEFs.
RankZhang and Pham [13]Zhu et al. [20]New1f1 (program complexity)f8 (frequency of program specification change)f11 (requirements analysis)2f15 (programmer skill)f22 (testing effort)f15 (programmer skill)3f25 (testing coverage)f21 (testing environment)f12 (relationship of detailed design to requirement)4f22 (testing effort)f25 (testing coverage)f20 (human nature)5f21 (testing environment)f1 (program complexity)f22 (testing effort)6f8 (frequency of program specification change)f15 (programmer skill)f24 (testing methodologies)7f24 (testing methodologies)f6 (percentage of reused modules)f18 (program workload(stress))8f11 (requirements analysis)f12 (relationship of detailed design to requirement)f6 (percentage of reused modules)9f6 (percentage of reused modules)f24 (testing methodologies)f19 (domain knowledge)10f12 (relationship of detailed design to requirement)f19 (domain knowledge)f7 (programming language)


### 4.2. Comparison of Principal Components

As shown in Table18, Zhang et al. and Zhu et al. [20] selected the top 11 and 10 SDEFs, respectively. In this study, the top 13 SDEFs were selected for factor analysis. This study applied factor analysis to identify the four common factors. The new principal components of this study are well structured by the software development phase compared to other studies.Table 18
Comparison of principal components.
Zhang and Pham [13]Zhu et al. [20]NewComponentPrincipleFactor (name)ComponentPrincipleFactor (name)ComponentPrincipleFactor (name)C1Overallf21 (testing environment)f22 (testing effort)f5 (level of programming technologies)f12 (relationship of detailed design to requirement)PC1Overallf25 (testing coverage)f21 (testing environment)f22 (testing effort)f24 (testing methodologies)f12 (relationship of detailed design to requirement)f6 (percentage of reused modules)PCA1Testing abilityf24 (testing methodologies)f25 (testing coverage)f22 (testing effort)C2Testing efficiencyf24 (testing methodologies)f25 (testing coverage)f6 (percentage of reused modules)PC2Specification and knowledgef8 (frequency of specification chance)f19 (domain knowledge)PCA2Programmer abilityf15 (programmer skill)f19 (domain knowledge)f20 (human nature)f7 (programming language)f18 (program workload (stress))C3Requirements and specificationsf11 (requirements analysis)f8 (frequency of specification chance)PC3Program complexity and skill levelf15 (programmer skills)f1 (program complexity)PCA3Requirements and standardsf11 (requirements analysis)f12 (relationship of detailed design to requirement)f13 (work standards)C4Program and skill levelf15 (programmer skills)f1 (program complexity)PCA4Program utilityf2 (program categories)f6 (percentage of reused modules)


### 4.3. Comparison of Significant SDEFs within Each Development Phase

As shown in Table19, this study and Zhu et al. [20] used the Tukey method to group the four development phases into mean scores, and Zhang et al. [21] used the SNK multiple comparison test. This study and Zhang et al. [21] only comprised one final group; however, they became two separate groups: testing and general phase in Zhu et al. [20]. In this study, the analysis and design phase is the most important development phase in software development; however, the testing phase is the most important phase in the software development phase.Table 19
Comparison of final grouping.
PhaseGrouping from Zhang et al. [21]Grouping from Zhu et al. [20]New groupingMeanSNKMeanTukeyMeanTukeyGeneral5.24A4.722A5.497AAnalysis and design5.03A5.034AB5.545ATesting5.43A4.933AB5.247ACoding5.35A5.225B5.265AA comparison of significant environmental factors is shown in Table20. F6, the percentage of reused modules, is the most significant factor from Zhang et al. [21] and Zhu et al. [20] in the general phase; however, it is not the most significant factor in the new results in Table 20. In the analysis and design phase, f14, development management is one of the significant environmental factors. F18, program workload, and f20, human nature, are significant environmental factors in the coding phase. Moreover, f22, the testing effort, is also a significant environmental factor. The reasons for the difference in significant factors in the results of the three papers are due to the difference in working environment, time, and more, as described in Section 4.1.Table 20
Comparison of significant environment factors.
PhaseZhang et al. [21]Zhu et al. [20]Results from Korean companiesSignificant EFsP valueSignificant EFsP valueSignificant EFsP valueGeneralf1 Program complexity0.0001f4 Amount of programming effort0.0001——f6 Percentage of reused code0.0907f6 Percentage of reused code0.013Analysis and designf8 Frequency of program specification change0.0635f8 Frequency of program specification change0.006f14 Development management0.003f10 Design methodology0.0063f12 Relationship of detailed design to requirement0.014f13 Work standards0.0068Codingf17 Development team size0.0192f15 Programmer skill0.032f18 Program workload (stress)0.015f19 Domain knowledge0.0341f18 Program workload (stress)0.0001f20 Human nature0.023Testingf21Testing environment0.0001f23 Testing resource allocation0.001f22 Testing effort<0.001f24 Testing methodologies0.017f25 Testing coverage0.002


### 4.4. Comparison of the Percentage of the Time Allocation for Software Development Phase

In this study, the percentages of time allocation for software development (analysis, design, coding, testing) were 21.33%, 25.33%, 30.40%, and 22.94%, respectively. The percentage of time allocation comparisons is presented in Table21. Compared to Zhang and Pham [13] and Zhu et al. [20], the percentages of time allocation were 3.7% and 0.7% lower in the analysis phase and 5.6% and 3.6% lower in the coding phase, respectively; however, it was 7.3% and 5.3% higher in the design phase. The time allocation for the testing phase had a similar percentage. Overall, it is observed that there is a small difference in each phase, but it seems that the design phase is more important in Korean companies.Table 21
Comparison of the percentage of the time allocation.
PhaseZhang and Pham [13]Zhu et al. [20]In Korea companiesAnalysis25%22%21.33(%)Design18%20%25.33(%)Coding36%34%30.40(%)Testing21%24%22.94(%)


## 4.1. Comparison of SDEFs Ranking

Since the study uses different population samples from different countries in its three analyses, and the time of the studies is also different, the ranking results differ. It is interesting to see how the ranking of SDEFs according to the relative weighting method changes with time and how it varies according to country. The three results for the top 10 ranks are listed in Table17. As shown in Table 17, the results of Zhang and Pham [13] and Zhu et al. [20] are similar. This can be studied in detail by Zhu et al. [20]. The results of the 10 new ranks also had similar SDEFs. However, there are three different environmental factors: f20, human nature; f18, program workload; and f7 programming language. First, the working area, working time, vacation, holiday, salary, relationship with boss, and culture are different. In Korea, software engineers belong to several categories. The proportion of engineers who occupy the total economy is a part of the medium/large core product development departments, large IT companies, agencies, enterprises, national laboratories, online/mobile game developers, high-growth start-ups, and others. In the middle/large core product development departments, there are numerous high-class talents, and the working time is long, but the treatment is good. The “IT department” of a company is not a core department but rather a department that is responsible for the computerization of a company. Moreover, since agencies begin with price competition, rather than quality from the outset, it is not easy for engineers to receive good treatment. In the case of enterprises and national laboratories, the quality of human resources is high, but the intensity of work is not. However, remuneration is low. In the case of online/mobile game developers, experienced workers receive good treatment, but the average remuneration is not high, as the percentage of junior (1–3 years after graduation) engineers is high. In the case of “high-growth start-ups” in Korea where competition for securing high-quality talent is fierce, treatment is good—this means that developers are working hard to work in a pleasant environment. In comparison to the US Silicon Valley, the proportion of large IT companies, agencies, and online/mobile game developers is relatively low in Korea. IT/infrastructure management is outsourced to low-cost markets, such as India or China, or resorts to cloud services. On the other hand, high value-added industries, which correspond to the medium/large core product development departments, and high-growth start-ups of large enterprises are adequately developing. F11, requirements analysis, is the most important environmental factor in the software development process. The customer’s knowledge level has increased to the extent that the program specifications and other factors vary according to customer requirements. In Korea, consumer requirements are changing faster than in the past. In addition, Korea has superior skills in IT-related industries compared to other countries.Table 17
Comparison of the ranking of SDEFs.
RankZhang and Pham [13]Zhu et al. [20]New1f1 (program complexity)f8 (frequency of program specification change)f11 (requirements analysis)2f15 (programmer skill)f22 (testing effort)f15 (programmer skill)3f25 (testing coverage)f21 (testing environment)f12 (relationship of detailed design to requirement)4f22 (testing effort)f25 (testing coverage)f20 (human nature)5f21 (testing environment)f1 (program complexity)f22 (testing effort)6f8 (frequency of program specification change)f15 (programmer skill)f24 (testing methodologies)7f24 (testing methodologies)f6 (percentage of reused modules)f18 (program workload(stress))8f11 (requirements analysis)f12 (relationship of detailed design to requirement)f6 (percentage of reused modules)9f6 (percentage of reused modules)f24 (testing methodologies)f19 (domain knowledge)10f12 (relationship of detailed design to requirement)f19 (domain knowledge)f7 (programming language)


## 4.2. Comparison of Principal Components

As shown in Table18, Zhang et al. and Zhu et al. [20] selected the top 11 and 10 SDEFs, respectively. In this study, the top 13 SDEFs were selected for factor analysis. This study applied factor analysis to identify the four common factors. The new principal components of this study are well structured by the software development phase compared to other studies.Table 18
Comparison of principal components.
Zhang and Pham [13]Zhu et al. [20]NewComponentPrincipleFactor (name)ComponentPrincipleFactor (name)ComponentPrincipleFactor (name)C1Overallf21 (testing environment)f22 (testing effort)f5 (level of programming technologies)f12 (relationship of detailed design to requirement)PC1Overallf25 (testing coverage)f21 (testing environment)f22 (testing effort)f24 (testing methodologies)f12 (relationship of detailed design to requirement)f6 (percentage of reused modules)PCA1Testing abilityf24 (testing methodologies)f25 (testing coverage)f22 (testing effort)C2Testing efficiencyf24 (testing methodologies)f25 (testing coverage)f6 (percentage of reused modules)PC2Specification and knowledgef8 (frequency of specification chance)f19 (domain knowledge)PCA2Programmer abilityf15 (programmer skill)f19 (domain knowledge)f20 (human nature)f7 (programming language)f18 (program workload (stress))C3Requirements and specificationsf11 (requirements analysis)f8 (frequency of specification chance)PC3Program complexity and skill levelf15 (programmer skills)f1 (program complexity)PCA3Requirements and standardsf11 (requirements analysis)f12 (relationship of detailed design to requirement)f13 (work standards)C4Program and skill levelf15 (programmer skills)f1 (program complexity)PCA4Program utilityf2 (program categories)f6 (percentage of reused modules)


## 4.3. Comparison of Significant SDEFs within Each Development Phase

As shown in Table19, this study and Zhu et al. [20] used the Tukey method to group the four development phases into mean scores, and Zhang et al. [21] used the SNK multiple comparison test. This study and Zhang et al. [21] only comprised one final group; however, they became two separate groups: testing and general phase in Zhu et al. [20]. In this study, the analysis and design phase is the most important development phase in software development; however, the testing phase is the most important phase in the software development phase.Table 19
Comparison of final grouping.
PhaseGrouping from Zhang et al. [21]Grouping from Zhu et al. [20]New groupingMeanSNKMeanTukeyMeanTukeyGeneral5.24A4.722A5.497AAnalysis and design5.03A5.034AB5.545ATesting5.43A4.933AB5.247ACoding5.35A5.225B5.265AA comparison of significant environmental factors is shown in Table20. F6, the percentage of reused modules, is the most significant factor from Zhang et al. [21] and Zhu et al. [20] in the general phase; however, it is not the most significant factor in the new results in Table 20. In the analysis and design phase, f14, development management is one of the significant environmental factors. F18, program workload, and f20, human nature, are significant environmental factors in the coding phase. Moreover, f22, the testing effort, is also a significant environmental factor. The reasons for the difference in significant factors in the results of the three papers are due to the difference in working environment, time, and more, as described in Section 4.1.Table 20
Comparison of significant environment factors.
PhaseZhang et al. [21]Zhu et al. [20]Results from Korean companiesSignificant EFsP valueSignificant EFsP valueSignificant EFsP valueGeneralf1 Program complexity0.0001f4 Amount of programming effort0.0001——f6 Percentage of reused code0.0907f6 Percentage of reused code0.013Analysis and designf8 Frequency of program specification change0.0635f8 Frequency of program specification change0.006f14 Development management0.003f10 Design methodology0.0063f12 Relationship of detailed design to requirement0.014f13 Work standards0.0068Codingf17 Development team size0.0192f15 Programmer skill0.032f18 Program workload (stress)0.015f19 Domain knowledge0.0341f18 Program workload (stress)0.0001f20 Human nature0.023Testingf21Testing environment0.0001f23 Testing resource allocation0.001f22 Testing effort<0.001f24 Testing methodologies0.017f25 Testing coverage0.002


## 4.4. Comparison of the Percentage of the Time Allocation for Software Development Phase

In this study, the percentages of time allocation for software development (analysis, design, coding, testing) were 21.33%, 25.33%, 30.40%, and 22.94%, respectively. The percentage of time allocation comparisons is presented in Table21. Compared to Zhang and Pham [13] and Zhu et al. [20], the percentages of time allocation were 3.7% and 0.7% lower in the analysis phase and 5.6% and 3.6% lower in the coding phase, respectively; however, it was 7.3% and 5.3% higher in the design phase. The time allocation for the testing phase had a similar percentage. Overall, it is observed that there is a small difference in each phase, but it seems that the design phase is more important in Korean companies.Table 21
Comparison of the percentage of the time allocation.
PhaseZhang and Pham [13]Zhu et al. [20]In Korea companiesAnalysis25%22%21.33(%)Design18%20%25.33(%)Coding36%34%30.40(%)Testing21%24%22.94(%)


## 5. Conclusions and Remarks

Software systems that offer user convenience must improve solutions for immediate problems in multiple industries. When using software for personal use, the schedule or quality of development is not a concern. However, good software for sale to the general public should be user-friendly, error-free, efficient, reliable, and utilitarian to satisfy its users. In many software development projects, software requirements are frequently modified during design or development, during which the development of specific designers’ or developers’ capabilities is tested. SDEFs, such as differences in mutual work recognition between users, developers, and testers or knowledge difference, can hinder communication, which may lead to faulty development due to erroneous job definition. This study analyzes the degree of impact of each SDEF on software reliability assessment in Korean companies. This study investigates 32 SDEFs to examine the impact of factors affecting the software development environment of Korean companies. The results were then compared to those of Zhang and Pham [13], Zhang et al. [21], and Zhu et al. [20]. It is worth noting that these three SDEFs, human nature (rank #4), program workload (rank #7), and programming language (#10) are in the top 10 rankings for the first time based on our survey data collected from Korean companies but have never been included among the top 10 in previous studies from US companies. These findings can provide useful benefits to software developers and managers who are working in countries with different or similar cultures, and how to allocate resources to identify up to date significant SDEFs in software development to increase working efficiency, i.e., work versus time investment in software reliability improvement.Briefly summarizing the results,(1)
Data were collected from formal survey questionnaires and Internet surveys, given directly by software developers or managers, in 11 organizations. Data from 75 surveys were used in the analysis.(2)
For Korean companies, the normalized weight value of factor f11, requirements analysis, is the highest, followed by f15, programmer skill; f12, relationship of detailed design to requirement; f20, human nature; and f22, testing effort.(3)
We found four principal components using factor analysis. The first component, PCA1, consisted of factors f24, f25, and f22, which are related to testing ability, whereas PCA2 consists of factors f15, f19, f20, f7, and f18, which are related to programmer ability. PCA3 consisted of factors f11, f12, and f13, whereas PCA4 consisted of factors f2 and f6, which were related to requirements, standards, and program utility, respectively. Cronbach’sα is 0.786, which is reliable because it is greater than 0.6.(4)
We performed ANOVA on hypotheses 1 to 5. As per the results of Hypothesis1, there was no significant difference according to the group. As per the results of Hypothesis 2, there is a significant difference only in the group of program utility component factors. A post-hoc analysis was performed using Scheffe’s method; however, there was no significant difference between the groups. As per the result of Hypothesis 3, there is a significant difference only in the programmer ability component factor group. The results of the post-hoc analysis showed a significant difference between groups of less than 5 years and more than 11 years. As per the results of Hypothesis 4, there is a significant difference only in the group of program utility component factors. In post-hoc analysis, there was no significant difference between the groups. According to the results of Hypothesis 5, there is a significant difference in the percentage of reused code. Post-hoc analysis showed significant differences between groups of less than 5 years and 5 to 10 years, and less than 5 years and more than 11 years.(5)
As a result of regression analysis to determine whether the four constituent factors affect the software reliability accuracy, it was confirmed that testing ability and programmer ability influence the improvement of software reliability accuracy.(6)
In the analysis and design phase, f14, development management is one of the significant environmental factors. F18, program workload, and f20, human nature, are significant environmental factors in the coding phase. Moreover, f22, the testing effort, is also a significant environmental factor. However, in the general phase, there were no significant environmental factors. F18, program workload; f20, human nature; and f22, testing effort, have high rankings based on the relative weighted method described in Table3.We compared the results of this study with those of previous studies. First, there are three environmental factors that differ from previous studies: f20, human nature, f18, program workload, and f7, programming language. These factors are influenced by the working area, working time, vacation, holiday, salary, relationship with bosses, and culture. Second, in the past two papers, testing plays a more important role; however, this study shows that analysis and design play a more important role than other phases. Third, as a result of comparing the significant environmental factors, the reason for the difference in significant factors in the results of the three papers is due to the difference in working environment, time, and more. Finally, we compared the time allocation for the software development phase. Overall, it can be observed that there is a small difference in each phase, but it seems that the design phase is more important in Korean companies. These results provide general guidance on important factors to consider for software developers and managers working in a culture or environment similar to a Korean company.



---

*Source: 6689905-2021-05-03.xml*