# Mobile Application Testing in Pakistan: A Survey

Muhammad Saqib Nawaz, Zafar Iqbal, Muhammad Ikramullah Lali, Raza Ul

► To cite this version:

Muhammad Saqib Nawaz, Zafar Iqbal, Muhammad Ikramullah Lali, Raza Ul. Mobile Application Testing in Pakistan: A Survey. Mehran University Research Journal of Engineering and Technology, 2018, 37 (1), pp.15-28. [https://hal.science/hal-01674993](https://hal.science/hal-01674993)

HAL Id: hal-01674993

https://hal.science/hal-01674993

Submitted on 3 Jan 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Mobile Application Testing in Pakistan: A Survey

MUHAMMAD SAQIB NAWAZ*, ZAFAR IQBAL**, MUHAMMAD IKRAMULLAH LALI***, AND RAZA UL MUSTAFA**

RECEIVED ON 23.06.2016 ACCEPTED ON 22.11.2016

## ABSTRACT

Research regarding MAT (Mobile Application Testing) in Pakistan is hard to discover and to the best of our knowledge, no work has been done in surveying MAT in Pakistan. In this work, we have examined the current trend and status of MAT in Pakistan. Main objective was to investigate to what extent MAT is currently applied in Pakistan software companies and what experience the companies have with using MAT. Furthermore, efforts were made to find out what testers think about MAT, e.g. issue, advantages and disadvantages of MAT, what factors affects MAT and how they plan to improve MAT. In order to achieve our objectives, we used a comprehensive online survey so we converted our research questions into correspondence survey questions. We served a questionnaire of the survey to 66 testing relevant officials of leading software companies in different cities of Pakistan to develop a model study about general trend and status of MAT which can be generalized all over Pakistan. We received 56 replies in total after over 2 months. After that, we used SPSS tool to analyze the replies of this questionnaire. Cross-Tabulation Analysis and Pearson Chi-square tests have been computed to examine the results. We found some interesting results on current status and practice of MAT in Pakistan software companies.

**Key Words:** Mobile Applications, Survey, Statistical Analysis, Testing.

## 1. INTRODUCTION

The future of computing is moving towards mobile devices with rapid advancements in ICT (Information & Communication Technology). The applications which are running on mobile devices are known as mobile applications. Mobile applications are becoming very famous as these are showing revolution in the field of information technology. Initially, mobile applications were developed mostly in the entertainment sector; however, they are now also available in other domains such as media, games, retail,

travel, education, healthcare, finance and social media platforms. In 2012, there were around 1.1 Billion users of mobile applications. Based on the forecast, the above said number will raise up quickly to almost 30% annually to get in touch with 4.4 Billion in 2017 [1]. So this huge demand for mobile devices, along-with mobile applications requires better mobile application testing tools [2].

In software quality assurance activities, testing is considered a key part in the life cycle of a software creation. Testing accomplish following two prime purposes [3]:

Corresponding Author (E-Mail: msaqibnawaz@pku.edu.cn)

* Department of Information Science, Peking University, Beijing, China.

** Department of Computer Science, University of Lahore, Sargodha Campus.

*** Department of Computer Science & Information Technology, University of Sargodha.
* It expresses the quality or accurate behavior.

* It identifies and fixes the problem.

Mobile testing refers to different types of testing, such as native and Web mobile application testing and testing of mobile devices [6]. We use MAT to refer to “testing activities for mobile applications (native, hybrid and Web applications) on mobile devices with software test methods and tools in order to ensure the correctness of functionality, behaviors, quality of service, usability, security and privacy of mobile applications”. Native mobile applications are coded in specific programming language (Java for Android platform and Object C for iOS) and executed on mobile devices. Native applications depend on device features, such as camera, contact list, GPS (Global Positioning System) and so on. Web mobile applications are run by a browser and generally written in HTML5 (Hyper Text Markup Language). It consists of an application server and client software. Users can access application through client software executed over browsers. Hybrid application is the combination of both the native and web applications. Hybrid is basically a web application that is built using HTML5 and JavaScript. It is then wrapped in a thin native container that offers access to device features.

MAT is currently applied in many countries. However, actual facts and figures regarding MAT in Pakistan software companies are not collected efficiently. In this work, we examined the current trend and status of MAT in Pakistan software companies. Our main focus is on the following research question:

*How to optimize the performance of MAT in Pakistani software companies?*

The aim is to explore the benefits of MAT as well as to improve the quality and promote MAT in Pakistan. Objective is to examine that how much MAT is performed

in Pakistani software companies and what advantages the companies have of using MAT. We have also examined affiliation among the roles, professional experience, history (age), location and size of the company.

We have collected the data from software companies located in different cities including ISD (Islamabad), SGD (Sargodha), LHR (Lahore), FSD (Faisalabad), Multan and Bahawalpur. For this purpose, a comprehensive online survey is conducted to collect responses from testing related persons. Rest of the article is organized as follows: Related work is discussed in Section 2. Our methodology, survey design and execution are stated in Section 3. Data collection from the survey, demographic information obtained from survey and main findings are presented in Section 4. Discussion and survey validity is discussed in Section 5, followed by conclusion in Section 6.

## 2. RELATED WORK

In the IT industry and in academia, MAT has recently received more attention by testing professionals due to increase in the number of testing tools and testing magazines are publishing issues exclusively on MAT [7]. Literature on MAT concepts, testing techniques, issues and challenges can be found in [4,6,8,9].

Yin [10] conducted a survey (executed in Finland and Estonia in the early of 2014) regarding software exploratory testing [11]. ET (Exploratory Testing) is a concurrent learning process in which the design and execution of the tests are not declared earlier. Design of software tests, execution of software tests and modification of software tests are made at run time. The objective and main research question of the survey [16] was to know the trend and status of existing practice about ET in Finland and Estonia. The questionnaire of the survey on software ET was served to 61 different roles such as developers, testers and test managers in various software companies in Finland and Estonia. Main findings of the survey were:
* Mostly testers (48%) and test managers (36%) apply ET in Estonian and Finnish companies.

* Approximately half of the respondents (50%) have more than five years of working experiences.

* Testers in Estonia (50%) use ET more than Finnish testers (38%).

* In small companies, ET has been applied relatively more often (less than 50 employees, 56%) than in large companies (32%).

* Companies older than 5 years that use ET are over 50%. Approximately 76% of the respondents who use ET are members of a separate testing organization within their company.

Madhushani et. al. [12] presented a mobile application testing procedure for Sri Lankan software development companies. They have also explained main challenges in MAT from Sri Lankan perspective. A questionnaire of 10 questions was distributed among various IT individuals and software companies in Sri Lanka. According to analysis:

* 93% IT industrials believes that MAT is very important for the customers.

* In Sri Lanka, 92% developers develop Android mobile application and 54% develop iOS based mobile application.

* Considering the level of testing, most of QA(Quality Assurance) engineers or developers use performance and security testing. Furthermore, few companies develop and test same application for different operating systems and devices.

* 92% testers use open source tools for the testing.

* Main challenges in MAT are variety of mobile devices and different operating systems.

### 3. METHODOLOGY

We have done a qualitative research and conducted an online survey from October 25, 2015 to December 31, 2015 in order to investigate the current status of MAT in Pakistan. Main objective was to investigate to what extent MAT is currently applied in Pakistan software companies and what experience the companies have with using MAT. Furthermore, we aimed to find out what factors affects MAT, such as specific roles in a company use MAT more often than others, and whether engineer experience, company size, company age, and location has any influence on the usage of MAT. In addition, what are the characteristics of MAT according to engineers and what they think about MAT, e.g. advantages and disadvantages of MAT and how they plan to improve MAT. Our main RQ is formulated to 3 research questions and each research question is further extended into related set of SQs (Survey Questions):

**RQ-1:** What are the abilities of those who perform MAT in Pakistan and what are the benefits of MAT for a company?

* **RQ-1.1:** Which mobile platform engineer prefer for mobile application development and testing?

* **RQ-1.2:** What are the main characteristics of mobile application?

* **RQ-1.3:** What type of mobile applications SW (Software) engineers develop and test?

* **RQ-1.4:** What types of tools are preferred by sw engineers to use in MAT?

* **RQ 1.5:** What is sw engineer trend to perform different types of testing?

* **RQ-1.6:** In what circumstances, sw engineer uses MAT for their company?

* **RQ-1.7:** What are the key advantages to the company of using MAT?
**RQ-2:** To what extent MAT is presently performed in Pakistani software companies and what are the influential factors on the usage of MAT in Pakistanis software companies?

**RQ-2.1:** Do testers perform MAT much commonly as compare to developers or programmers etc?

**RQ-2.2:** Do sw engineer having much professional experience perform MAT much commonly?

**RQ-2.3:** Does the localities (SGD, LHR, ISD and FSD) have different outcomes on performing MAT?

**RQ-2.4:** Do employees of big company perform MAT much frequently as compare to the small companies?

**RQ-2.5:** Is MAT performed much commonly in old companies?

**RQ-2.6:** Do engineers of separate testing organization perform MAT much commonly?

**RQ-2.7:** What factors are dissimilar when performing MAT in SGD, LHR, ISD and FSD?

**RQ-3:** What are the opinions of SW engineers regarding MAT and how improvements can be brought in existing practice of MAT in Pakistan?

**RQ-3.1:** What are the basics, sw engineers think as necessary for defining MAT?

**RQ-3.2:** What uniqueness do sw engineers feel about MAT?

**RQ-3.3 and 3.4:** What are the opinions of sw engineers regarding advantages and disadvantages of MAT?

**RQ-35 and 3.6:** What are the issues in MAT and what are the key steps for improvement in existing practice of MAT in Pakistan?

From these three RQ's, we have designed SQ's. SQ's and their mapping to RQ's is shown in Appendix-A.

## 4. SURVEY DISTRIBUTION AND RESULTS

Online Survey questionnaire was served to various mobile applications developers, testers, managers and other test relevant persons that cooperate with managers or testers. For this purpose, we choose Google Docs platform to serve and receive online replies from respondents. We have also posted the survey link through other channels such as mailing list, LinkedIn, Twitter and Facebook. We collected 56 online replies of our survey questionnaire. The results of data collection of survey questionnaire in graphical representations are discussed next.

### 4.1 Demographic Information

The demographic information of survey respondents is shown in Figs. 1-6. Figures in the graphs show frequencies and percentage (for example, 24, 46% means that 24 respondents of survey select that choice and its percentage is 46). Respondents' role and the total working experience of the respondent in current organization is shown in Figs. 1-2 respectively. 73% respondents are either working as manager or developer in the company. Whereas, 9 out of 56 respondents (16%) are working as testers. Most of the respondents (41 out of 56) have more than 3 years of working experience in their current role and 15 respondents (27%) have less than 3 years of working experience. Company location, size and age of the company is shown in Figs. 3-5 respectively. In 56 responses, 49 respondents belong to four cities (LHR, FSD, ISD and SGD) whereas 7 respondents belong to other cities (Multan and Bahawalpur). The characteristics of the mobile applications that respondents' organization deals with are shown in Fig. 6. The most frequently mentioned mobile application characteristics is 'high performance' (30%), followed by 'high security' (23%) and 'complex GUI' (18%).
## 4.2 Main Findings

We used SPSS [14] tool to analyze the replies of this questionnaire. Statistical tests such as Cross-Tabulation, Fisher Exact Tests and Pearson Chi-Square Tests [5,15] have been computed to examine the results. Statistical tests give the information that can be used to assess the significances of increases or decreases in the results. Correct selection of statistical test depends on data type as well as aim of the analysis. In case of wrong selection of statistical test, required results may not be valid and can mislead to conclusion of the research. In this section, survey results relevant to each RQ are reported.

<figure><img src="image_2.png" /><figcaption>FIG. 1. RESPONDENTS' ROLE</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>FIG. 2. RESPONDENTS' WORKING EXPERIENCE</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>FIG. 3. RESPONDENTS' COMPANY LOCATION</figcaption></figure>

### 4.2.1 RQ-1

What are the abilities of those who perform MAT in Pakistan and what are the benefits for their companies of MAT?

In the following subsections, results related to RQ-1 for each sub-question is reported.

**RQ-1.1** Which mobile platform engineer prefer for mobile application development and testing?

Table 1 shows the distribution of mobile platforms that survey respondents' use for mobile application development and testing. 54 out of 56 (96%) respondents'

<figure><img src="image_5.png" /><figcaption>FIG. 4. RESPONDENTS' COMPANY SIZE</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>FIG. 5. AGE OF RESPONDENTS' COMPANY</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>FIG. 6. MOBILE APPLICATION CHARACTERISTICS</figcaption></figure>
develop and test mobile application in Android and iOS. This is because 82% mobile users in Pakistan use Android based Mobiles and 12.9% use iOS, according to online report [13].

**RQ-1.2:** What are the main characteristics of mobile application?

Main characteristics of mobile applications according to survey respondents' are listed in Table 2. 30% respondents (17 out of 56) believe that 'high performance' is the main characteristic of their application. 13 respondents (23%) said that 'high security and safety' is the main characteristics of mobile applications that they developed and test.

**RQ-1.3:** What type of mobile applications SW engineers develop and test?

Table 3 lists the distribution of mobile applications that survey respondents' generally develop and test in their company. Almost 80% engineer test either web application or hybrid application.

**RQ-1.4:** What types of tools are preferred by SW engineers to use in MAT?

From RQ-1.1, 54 respondents (98%) said that either use Android or iOS platform for mobile application development and testing. So we have asked respondents' that what types of tools they prefer for MAT in Android and iOS. In Table 4, tools used by engineers for Android and iOS application testing is listed.

<table><caption>TABLE 1. DISTRIBUTION OF MOBILE PLATFORMS FOR DEVELOPMENT AND TESTING</caption><thead><tr><td>Mobile Platform</td><td>Sargodha</td><td>Lahore</td><td>Faisalabad</td><td>Islamabad</td><td>Other City</td><td>Total</td></tr></thead><tbody><tr><td>Android</td><td>6 (16.5%)</td><td>10 (28%)</td><td>6 (16.5%)</td><td>9 (25%)</td><td>5 (14%)</td><td>36 (64%)</td></tr><tr><td>Blackberry</td><td>0</td><td>1 (100%)</td><td>0</td><td>0</td><td>0</td><td>1 (2%)</td></tr><tr><td>Windows</td><td>0</td><td>0</td><td>0</td><td>1 (100%)</td><td>0</td><td>1 (2%)</td></tr><tr><td>iOS</td><td>3 (16.5%)</td><td>5 (27.5%)</td><td>5 (27.5%)</td><td>3 (16.5%)</td><td>2 (11%)</td><td>18 (32%)</td></tr><tr><td>Total</td><td>9 (16%)</td><td>16 (29%)</td><td>11 (20%)</td><td>13 (23%)</td><td>7 (12%)</td><td>56</td></tr></tbody></table>

<table><caption>TABLE 2. MAIN CHARACTERISTICS OF MOBILE APPLICATIONS</caption><thead><tr><td>Characteristics</td><td>Frequency</td><td>Percent</td></tr></thead><tbody><tr><td>High Security and Safety</td><td>13</td><td>23</td></tr><tr><td>Complex GUI</td><td>10</td><td>18</td></tr><tr><td>High performance</td><td>17</td><td>30</td></tr><tr><td>Energy Saving</td><td>7</td><td>13</td></tr><tr><td>Other</td><td>9</td><td>16</td></tr><tr><td>Total</td><td>56</td><td></td></tr></tbody></table>

<table><caption>TABLE 3. DISTRIBUTION OF MOBILE APPLICATIONS SURVEY CORRESPONDENTS' DEVELOP AND TEST</caption><thead><tr><td>Mobile App Type</td><td>Sargodha</td><td>Lahore</td><td>Faisalabad</td><td>Islamabad</td><td>Other City</td><td>Total</td></tr></thead><tbody><tr><td>Native App</td><td>2 (16.5%)</td><td>3 (25%)</td><td>2 (16.5%)</td><td>3 (25%)</td><td>2 (17%)</td><td>12 (22%)</td></tr><tr><td>Hybrid App</td><td>3 (13%)</td><td>7 (32%)</td><td>5 (23%)</td><td>4 (18%)</td><td>3 (13.5%)</td><td>22 (39%)</td></tr><tr><td>Web App</td><td>4 (17.5%)</td><td>6 (27.5%)</td><td>4 (17.5%)</td><td>6 (27.5%)</td><td>2 (9%)</td><td>22 (39%)</td></tr><tr><td>Total</td><td>9 (16%)</td><td>16 (29%)</td><td>11 (20%)</td><td>13 (23%)</td><td>7 (12%)</td><td>56</td></tr></tbody></table>
13 out of 36 respondents' (36%) use SOASTA tool to perform MAT in Android platform. Whereas, 44% respondents (8 out of 18) used Ranorex tool for iOS platform.

**RQ-1.5:** What is SW engineer trend to perform different types of testing.

Data collection from SQ-1.10 and SQ-2.3 shows that system testing (25 out of 56) is performed more times as compare to other types of testing. On the other hand acceptance testing is performed very less (9 out of 56). Survey results for RQ-1.5 are shown in Fig. 7.

**RQ-1.6:** In what circumstances, SW engineer uses MAT for their company?

Testing of mobile application is generally done during application development or after the development. Almost half of respondents (27 out of 56) perform MAT during the development and other half performs MAT after application development (as shown in Fig. 8). In order to check whether there is any difference by applying MAT during application development or after the development, we stated following null hypothesis.

<figure><img src="image_8.png" /><figcaption>FIG. 8. RESPONSES FROM RESPONDENTS REGARDING TESTING TIME OF MAT</figcaption></figure>

<table><caption>TABLE 4. TOOLS USED FOR ANDROID AND IOS APPLICATIONS TESTING</caption><thead><tr><td>Tools for MAT</td><td>Android</td><td>iOS</td><td>Total</td></tr></thead><tbody><tr><td>Perfecto Mobile</td><td>3 (8%)</td><td>2 (11%)</td><td>5 (9%)</td></tr><tr><td>Device Anywhere</td><td>6 (17%)</td><td>3 (17%)</td><td>9 (17%)</td></tr><tr><td>SOASTA</td><td>13 (36%)</td><td>1 (5%)</td><td>14 (26%)</td></tr><tr><td>JamoMuex</td><td>2 (5%)</td><td>1 (5%)</td><td>3 (5%)</td></tr><tr><td>ZapFix</td><td>2 (6%)</td><td>2 (11%)</td><td>4 (7%)</td></tr><tr><td>Ranorex</td><td>8 (22%)</td><td>8 (44%)</td><td>16 (30%)</td></tr><tr><td>No Tool</td><td>2 (6%)</td><td>1 (6%)</td><td>3 (6%)</td></tr><tr><td>Total</td><td>36</td><td>18</td><td>54</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>FIG. 7. TRENDS TO PERFORM DIFFERENT TYPES OF TESTING</figcaption></figure>
H0: There is no difference of performing MAT with respect to time.

H1: There is a difference of performing MAT with respect to time.

To test these hypotheses, we used Pearson Chi square test. We used SPSS tool for all calculations and the value of significant level alpha (α) is set to 0.05. Table 5 shows the test results.

Table 5 shows that *p* = 0.891 > α (0.05), hence accept hypothesis *H0* (There is no difference of performing MAT with respect to time) and reject hypothesis *H1*.

**RQ-1.7:** What are the key advantages to the company of using MAT?

Table 6 shows that 19 out of 56 respondents (4 managers, 11 developers, 3 tester, 1 other) believes that most important benefit to the company of performing MAT is that 'the company produces high quality mobile application'. 15 respondents' believe that 'company gets the attention of customer' by using MAT.

**4.2.2. RQ-2:** To what extent MAT is presently performed in Pakistani software companies and what are influential factors on the usage of MAT in Pakistanis software companies?

For RQ-2.1 to RQ-2.6, we formulated and tested following null hypothesis:

[RQ-2.1] *H0-1:* There is a difference in frequencies of performing MAT for testers and developers.

[RQ-2.2] *H0-2:* There is no difference in performing MAT to dissimilar level of professional experience.

[RQ-2.3] *H0-3:* There is no difference of performing MAT in different cities.

[RQ-2.4] *H0-4:* There is no difference of performing MAT in different sizes of companies.

[RQ-2.5] *H0-5:* There is no difference of performing MAT in different ages of companies.

[RQ-2.6] *H0-6:* MAT performance is not affected by the existence of a separate testing company.

<table><caption>TABLE 5. PEARSON CHI SQUARE TEST RESULTS</caption><thead><tr><th></th><th>Value</th><th>df</th><th>Asymp. Significant (2-Sided)</th></tr></thead><tbody><tr><td>Pearson Chi-Square</td><td>0.622a</td><td>3</td><td>0.891</td></tr><tr><td>Likelihood Ratio</td><td>0.629</td><td>3</td><td>0.890</td></tr><tr><td>Linear-by-Linear Association</td><td>0.001</td><td>1</td><td>0.978</td></tr><tr><td>N of Valid Cases</td><td>56</td><td></td><td></td></tr></tbody><tfoot><tr><td colspan="4">a. 4 cells (50.0%) have expected count less than 5.</td></tr></tfoot></table>

<table><caption>TABLE 6. COMPANY BENEFITS BY USING MAT</caption><thead><tr><th>Benefits to Company</th><th>Manager</th><th>Developer</th><th>Tester</th><th>Other</th><th>Total</th></tr></thead><tbody><tr><td>Company meets the standards of mobile app development as market's demand</td><td>3 (27%)</td><td>5 (46%)</td><td>2 (18%)</td><td>1 (9%)</td><td>11 (20%)</td></tr><tr><td>Company gets the attention of customer</td><td>2 (14%)</td><td>9 (60%)</td><td>2 (13%)</td><td>2 (13%)</td><td>15 (27%)</td></tr><tr><td>Company produces high quality mobile applications.</td><td>4 (21%)</td><td>11 (58%)</td><td>3 (16%)</td><td>1 (5%)</td><td>19 (33%)</td></tr><tr><td>Company gets financial benefits.</td><td>1</td><td>3</td><td>1</td><td>1</td><td>6 (11%)</td></tr><tr><td>Other</td><td>1 (16%)</td><td>2 (50%)</td><td>1 (17%)</td><td>1 (17%)</td><td>5 (9%)</td></tr></tbody></table>
All the survey respondents' were familiar with MAT. Data related to RQ 2.1 and RQ 2.2 is listed in Table 7. Responses from survey respondents' regarding whether they work as separate tester are listed in Table 8. Survey respondents' company location, company size and company age is already shown in Figs. 3-5 respectively. For hypothesis testing of RQ-2, we performed Cross-tabulation analysis and Pearson Chi-square tests in SPSS. We set the significance level alpha (α) = 0.05. Test results are shown in Table 9.

It turns out that only hypothesis *H0-1* could be rejected. There is no difference in frequencies of performing MAT for testers.

**RQ-2.7:** What factors are dissimilar when testers perform MAT among SGD, LHR, ISD and FSD?

Some dissimilar factors were also found in survey research as results of SQ-1.1 and SQ-1.3 shows that 30% of respondents of ISD are managers of software companies.

There is no professional tester found in FSD in our survey. Mostly developers and managers perform testing of applications by themselves. Results of SQ-1.3, SQ-1.4 and SQ-1.5 show that biggest and oldest companies are located in ISD. Results of SQ-1.2 show that most experienced developers and managers also belong to ISD.

**4.2.3. RQ-3:** What are the opinions of testers regarding MAT and how improvements can be brought in existing practice of MAT in Pakistan?

**RQ-3.1:** What are the basics, SW engineers think as necessary for defining MAT?

In RQ-3.1, we asked respondents to define MAT by selecting three choices. Result is shown in Table 10. 52 out of 56 respondents (31%) think that 'trained and professional team and skill' are the basics of MAT. 36 respondents (21%) have the opinion that "quality assurance methodologies for MAT" are the basics.

<table><caption>TABLE 7. RESPONSES FOR RQ-2.1 AND RQ-2.2</caption><thead><tr><td>Do You or Your Colleagues Perform MAT</td><td>Manager</td><td>Developer</td><td>Tester</td><td>Other</td><td>Total</td></tr></thead><tbody><tr><td>I perform MAT</td><td>6 (24%)</td><td>10 (40%)</td><td>6 (24%)</td><td>3 (12%)</td><td>25 (45%)</td></tr><tr><td>I perform MAT and my colleagues also perform.</td><td>3 (14%)</td><td>16 (76%)</td><td>1 (5%)</td><td>1 (5%)</td><td>21 (37%)</td></tr><tr><td>I perform MAT but my colleagues do not perform.</td><td>2 (20%)</td><td>4 (40%)</td><td>2 (20%)</td><td>2 (20%)</td><td>10 (18%)</td></tr><tr><td>Total</td><td>11 (20%)</td><td>30 (53%)</td><td>9 (16%)</td><td>6 (11%)</td><td>56</td></tr></tbody></table>

<table><caption>TABLE 8. RESPONDENT WORKS AS SEPARATE TESTER</caption><thead><tr><td>Respondent Works as Separate Tester</td><td>Frequency</td><td>Percent</td></tr></thead><tbody><tr><td>Yes</td><td>38</td><td>68%</td></tr><tr><td>No</td><td>18</td><td>32%</td></tr><tr><td>Total</td><td>56</td><td>100%</td></tr></tbody></table>

<table><caption>TABLE 9. TEST RESULTS FOR RQ-2.1 TO RQ-2.6</caption><thead><tr><td>Chi-square Tests</td><td>RQ-2.1</td><td>RQ-2.2</td><td>RQ-2.3</td><td>RQ-2.4</td><td>RQ-2.5</td><td>RQ-2.6</td></tr></thead><tbody><tr><td>Df</td><td>6</td><td>4</td><td>8</td><td>4</td><td>4</td><td>2</td></tr><tr><td>p-Value</td><td>0.157</td><td>0.350</td><td>0.694</td><td>0.885</td><td>0.947</td><td>0.556</td></tr><tr><td>N</td><td>56</td><td>56</td><td>56</td><td>56</td><td>56</td><td>46</td></tr></tbody></table>
**RQ-3.2:** What uniqueness do SW engineers feel about MAT?

We asked respondents what uniqueness they fell about MAT by selecting three choices. Based on the results (shown in Table 11) of SQ-1.1 and SQ-3.2, 'mobile application testing is efficient' is the highly appreciated characteristics of MAT.

**RQ-3.3 and 3.4:** What is the opinion of testers regarding advantages and advantages of MAT?

In RQ-3.3, and RQ-3.4, we asked respondents to give their opinions on advantages and disadvantages of MAT by selecting at most 5 options (shown in Table 12). 31% respondents' believe that 'effectiveness and efficiency' are the main advantage of MAT. On the other hand, 26% respondents' have the opinion (shown in Table 13) that MAT is 'hard to report' and is 'inflexible'.

**RQ-3.5 and RQ-3.6:** What are the issues in MAT and what are the key steps for improvement in existing practice of MAT in Pakistan?

Data collection from SQ-3.5 (issues in MAT) is listed in Table 14. Out of 56 respondents, 34% believe that the main issue in MAT is its 'security'.

In RQ-3.6, we asked respondents to give their opinions in connection with the improvements in MAT by selecting 3 options. Results are listed in Table 15. 20% respondents believe that 'testers may be trained' in order to improve current status of MAT in Pakistan. 18% believes that 'adapt latest technology' in order to improve MAT.

## 5. DISCUSSION AND VALIDITY OF SURVEY

Results presented in previous section are discussed in this section. Furthermore, the most pressing issue of our study is also discussed.

<table><caption>TABLE 10. RESPONSES REGARDING MAT BASICS</caption><thead><tr><td>Basics of MAT</td><td>Total</td><td>Percent</td></tr></thead><tbody><tr><td>Trained and Professional Team and Skills</td><td>52</td><td>31</td></tr><tr><td>Quality Assurance methodologies for MAT</td><td>36</td><td>21</td></tr><tr><td>MAT tools</td><td>31</td><td>19</td></tr><tr><td>MAT technology vision</td><td>21</td><td>13</td></tr><tr><td>MAT development</td><td>14</td><td>8</td></tr><tr><td>MAT environment</td><td>9</td><td>5</td></tr><tr><td>Other</td><td>5</td><td>3</td></tr></tbody></table>

<table><caption>TABLE 11. MAT UNIQUENESS</caption><thead><tr><td>Characteristics of MAT</td><td>Manager</td><td>Developer</td><td>Tester</td><td>Other</td><td>Total</td></tr></thead><tbody><tr><td>MAT supports creativity</td><td>5 (20%)</td><td>15 (60%)</td><td>4 (16%)</td><td>1 (4%)</td><td>25 (15%)</td></tr><tr><td>MAT is focused</td><td>1 (12.5%)</td><td>6 (75%)</td><td>1 (12.5%)</td><td>0</td><td>8 (5%)</td></tr><tr><td>MAT is flexible</td><td>5 (22%)</td><td>13 (56%)</td><td>3 (13%)</td><td>2 (9%)</td><td>23 (14%)</td></tr><tr><td>MAT is efficient</td><td>11 (25%)</td><td>20 (46%)</td><td>9 (20%)</td><td>4 (9%)</td><td>44 (26%)</td></tr><tr><td>MAT is effective</td><td>6 (19%)</td><td>17 (55%)</td><td>3 (10%)</td><td>5 (16%)</td><td>31 (18%)</td></tr><tr><td>MAT is easy to use</td><td>3 (15%)</td><td>10 (50%)</td><td>3 (15%)</td><td>4 (20%)</td><td>20 (12%)</td></tr><tr><td>MAT makes testing interesting</td><td>2 (12%)</td><td>9 (53%)</td><td>4 (23%)</td><td>2 (12%)</td><td>17 (10%)</td></tr></tbody></table>
Regarding RQ-1, we found the following:

■ 64% software engineer uses Android platform and 32% uses iOS for mobile applications development and testing.

■ 30% (17 out of 56) respondents' believe that main characteristics of mobile application is 'high performance'.

■ 78% engineers develop and test hybrid and Web mobile applications.

<table><caption>TABLE 12. ADVANTAGES OF MAT</caption><thead><tr><td>Advantages of MAT</td><td>Manager</td><td>Developer</td><td>Tester</td><td>Other</td><td>Total</td></tr></thead><tbody><tr><td>Effective</td><td>7 (15%)</td><td>27 (56%)</td><td>9 (19%)</td><td>5 (10%)</td><td>48 (17%)</td></tr><tr><td>Emphasizes tester</td><td>5 (29%)</td><td>6 (35%)</td><td>4 (24%)</td><td>2 (12%)</td><td>17 (6%)</td></tr><tr><td>Efficient</td><td>9 (22%)</td><td>23 (58%)</td><td>6 (15%)</td><td>2 (5%)</td><td>40 (14%)</td></tr><tr><td>Support Creativity</td><td>6 (19%)</td><td>20 (62%)</td><td>5 (16%)</td><td>1 (3%)</td><td>32 (11%)</td></tr><tr><td>Supports Learning</td><td>3 (11%)</td><td>15 (54%)</td><td>6 (21%)</td><td>4 (14%)</td><td>28 (10%)</td></tr><tr><td>Time saving</td><td>6 (25%)</td><td>12 (50%)</td><td>3 (12%)</td><td>3 (13%)</td><td>24 (9%)</td></tr><tr><td>Easy</td><td>6 (22%)</td><td>13 (48%)</td><td>5 (19%)</td><td>3 (11%)</td><td>27 (10%)</td></tr><tr><td>Interesting</td><td>3 (15%)</td><td>11 (55%)</td><td>1 (5%)</td><td>5 (25%)</td><td>20 (7%)</td></tr><tr><td>Flexible</td><td>3 (22%)</td><td>6 (43%)</td><td>2 (14%)</td><td>3 (21%)</td><td>14 (5%)</td></tr><tr><td>Focused</td><td>2 (14%)</td><td>10 (72%)</td><td>1 (7%)</td><td>1 (7%)</td><td>14 (5%)</td></tr><tr><td>Independent</td><td>5 (31%)</td><td>7 (44%)</td><td>3 (19%)</td><td>1 (6%)</td><td>16 (6%)</td></tr></tbody></table>

<table><caption>TABLE 13. DISADVANTAGES OF MAT</caption><thead><tr><td>Disadvantages of MAT</td><td>Manager</td><td>Developer</td><td>Tester</td><td>Other</td><td>Total</td></tr></thead><tbody><tr><td>Inflexible</td><td>5 (15%)</td><td>18 (53%)</td><td>6 (17%)</td><td>5 (15%)</td><td>34 (12%)</td></tr><tr><td>Ineffective</td><td>6 (24%)</td><td>12 (48%)</td><td>4 (16%)</td><td>3 (12%)</td><td>25 (9%)</td></tr><tr><td>Inaccurate</td><td>3 (17%)</td><td>10 (59%)</td><td>1 (6%)</td><td>3 (18%)</td><td>17 (6%)</td></tr><tr><td>No focus</td><td>3 (15%)</td><td>12 (60%)</td><td>3 (15%)</td><td>2 (10%)</td><td>20 (7%)</td></tr><tr><td>Unnecessary</td><td>4 (31%)</td><td>7 (54%)</td><td>1 (7%)</td><td>1 (8%)</td><td>13 (5%)</td></tr><tr><td>Time limit</td><td>6 (19%)</td><td>17 (54%)</td><td>6 (19%)</td><td>3 (9%)</td><td>32 (11%)</td></tr><tr><td>Hard to report</td><td>9 (24%)</td><td>18 (47%)</td><td>7 (18%)</td><td>4 (11%)</td><td>38 (14%)</td></tr><tr><td>Hard to compare results</td><td>5 (21%)</td><td>14 (58%)</td><td>3 (13%)</td><td>2 (8%)</td><td>24 (9%)</td></tr><tr><td>Complicated</td><td>6 (20%)</td><td>16 (53%)</td><td>6 (20%)</td><td>2 (7%)</td><td>30 (11%)</td></tr><tr><td>Confusing</td><td>3 (17%)</td><td>11 (61%)</td><td>2 (11%)</td><td>2 (11%)</td><td>18 (6%)</td></tr><tr><td>Time consuming</td><td>5 (17%)</td><td>15 (52%)</td><td>6 (21%)</td><td>3 (10%)</td><td>29 (10%)</td></tr></tbody></table>

<table><caption>TABLE 14. RESPONSES REGARDING ISSUES IN MAT</caption><thead><tr><td>Issues in MAT</td><td>Total</td><td>Frequency (%)</td></tr></thead><tbody><tr><td>User experience</td><td>9</td><td>16</td></tr><tr><td>Coverage of available mobile hardware</td><td>16</td><td>29</td></tr><tr><td>Connectivity</td><td>12</td><td>21</td></tr><tr><td>Security</td><td>19</td><td>34</td></tr><tr><td>Total</td><td>56</td><td></td></tr></tbody></table>
* For Android based applications, SOASTA is the preferred testing tools for engineers whereas Ranorex is the preferred tool for iOS.

* There is no difference of performing MAT with respect to time. 48% engineers use testing during mobile application development and 52% uses it after mobile application development

* Main advantage of MAT for the company is that 'the company produces high quality mobile applications'.

Regarding RQ-2, we found the following:

* There is no difference in frequencies of performing MAT for sw engineers.

* There is no difference in performing MAT to dissimilar level of professional experience.

* There is no difference of performing MAT in different cities.

* There is no difference of performing MAT in different sizes of companies.

* There is no difference of performing MAT in different ages of companies.

* MAT performance is not affected by the existence of separate testing company.

* 30% of respondents of ISD are managers of software companies. There is no professional tester found in FSD in our survey. Mostly developers and managers perform testing of their applications themselves.

Regarding RQ-3, we found the following:

* More than half of the respondents' (52%) has the opinion that that 'trained and professional team and skill' and 'Quality Assurance methodologies' are the basics for defining MAT.

* MAT is 'effective' and 'efficient' (advantages) and it is 'hard to report' and 'inflexible' (disadvantages).

* Main issue in MAT is 'security' and 38% respondents believe that 'testers may be trained' and 'adapt latest technology' in order to improve MAT.

## 5.1. Validity of Survey

Every empirical study has shortcomings, so does our study too. Some issues related to our study are discussed next.

<table><caption>TABLE 15. RESPONSES ON IMPROVEMENTS IN MAT</caption><thead><tr><td>Improvements in MAT</td><td>Manager</td><td>Developer</td><td>Tester</td><td>Other</td><td>Total</td></tr></thead><tbody><tr><td>Maintain record for MAT</td><td>4(25%)</td><td>8(50%)</td><td>2(12%)</td><td>2(13%)</td><td>16(10%)</td></tr><tr><td>Select MAT methods according to app</td><td>3(16%)</td><td>10(52%)</td><td>3(16%)</td><td>3(16%)</td><td>19(11%)</td></tr><tr><td>Reports may be improved</td><td>3(27%)</td><td>5(46%)</td><td>3(27%)</td><td>0</td><td>11(7%)</td></tr><tr><td>Make reports more effective and efficient</td><td>2(22%)</td><td>5(56%)</td><td>1(11%)</td><td>1(11%)</td><td>9(5%)</td></tr><tr><td>Improve the testing process all the time</td><td>5(28%)</td><td>7(39%)</td><td>3(16%)</td><td>3(17%)</td><td>18(11%)</td></tr><tr><td>More risk-based testing</td><td>1(14%)</td><td>5(72%)</td><td>1(14%)</td><td>0</td><td>7(4%)</td></tr><tr><td>Use best tools</td><td>3(12%)</td><td>16(67%)</td><td>3(13%)</td><td>2(8%)</td><td>24(14%)</td></tr><tr><td>Adopt latest technology related to MAT</td><td>7(23%)</td><td>15(50%)</td><td>5(17%)</td><td>3(10%)</td><td>30(18%)</td></tr><tr><td>Testers may be trained for MAT</td><td>5(15%)</td><td>19(65%)</td><td>6(17%)</td><td>4(12%)</td><td>34(20%)</td></tr></tbody></table>
**Unfairness Selection:** Firstly, our research consists powerful pertinent. Our goal was fully concerned to the testing of mobile applications. This is why we could not approach the various respondents at initial stages. For instance, a developer or tester who is not familiar with MAT hears at a glance the survey regarding MAT; he/ she might want to ignore the survey as soon as possible. That is the main cause of achieving 100% encouraging replies (respondents knows about MAT earlier).

Secondly, we conducted our survey mostly in four cities (SGD, LHR, ISD and FSD) but we still obtained five replies other than above mentioned cities. In the meantime, we were not successful to get maximum responses due to restricted dispersion of the survey.

**External Validity:** An additional flaw of the survey is the small amount of replies. This shows a vast hazard to external soundness of the survey. More than 500 mobile applications developers or tester could be found in above mentioned four cities but we got only 56 replies. Moreover, we could not identify accurately how much percentage of all the mobile applications developers or testers of above said four cities we approached and how much is the rate of their responses.

**Internal Validity:** Strength of survey depends on design of survey questions. If research questions are precisely converted into survey questions and every one of the possibilities is enclosed by answer choices thus the quality of survey enhanced. It means, each survey question and its answer must be understandable by the respondents. So, to overcome the internal validity threats of our survey, we evaluated and reorganized our survey questions many times. For this purpose, we got help from experts/professors from academia and institutions to make our survey questionnaire easy, interesting, relevant to the topic, comprehensive as well as clear for respondents. Thus, we enhanced the survey based on time to time evaluation to move up its internal strength.

**Statistical Conclusion Validity:** There is a powerful link among reliability of statistical results, size and coverage of the sample. As quantity of sample increases, ultimately, broad coverage of sample also increases so statistical results becomes much valuable. In our survey, the size of the sample is not very big that is why there is a great threat to our research on the subject of validity of statistical results. This threat can be minimized in upcoming time by gathering a great number of samples.

## 6. CONCLUSION

In this work, an online survey is conducted to investigate the current status and trends of MAT in Pakistan and we found some interesting answers on the practice of MAT. In total, we received 56 replies from software engineers out of which 51 were from software companies located in Islamabad, Lahore, Faisalabad and Sargodha. It comes to our notice that there are few professional testers in all our dataset, this is why, most of the developers and testers even managers of software companies perform testing themselves. Results are very fruitful to know that what kind of tools, mobile platforms and the test strategies are considered as best for MAT along with advantages, disadvantages and how improvements can be made in current MAT practice in Pakistan so it can be valuable and proficient to managers, developer and testers as well as companies.

56 responses in total is not a very good statistic amount. In future we would like to implement the survey again and try to get more responses and include other big cities in our survey such as Karachi and Peshawar. Furthermore, in order to get a higher response rate and a more representative set of responses, we would like to have an agreement with some companies and make sure that employees of those companies answer our survey.
ACKNOWLEDGEMENT

Authors would like to thank all the respondents of survey
and several experts/professionals from academia who
helped in the design of our survey questionnaire.

REFERENCES

[1] Schweighofer, T., and Hericko, M., "Mobile Device and Technology Characteristics: Impact on Mobile Application Testing", In Proceedings of 2nd International Conference on Software Quality Analysis, Monitoring, Improvement and Applications, pp. 103-108, 2013.

[2] Merwe, H., Merwe, B., and Visser, W., "Verifying Android Applications using Java PathFinder", ACM SIGSOFT Software Engineering Notes, Volume 37, No. 3, pp. 1-5, 2012.

[3] Tian, J., "Software Quality Engineering: Testing, Quality Assurance, And Quantifiable Improvement", John Wiley & Sons Publishers, Hoboken, New Jersey, USA, 2005.

[4] Gao, J., Bai, X., Tsai, W.T., and Uehara, T., "Mobile Application Testing", Computer, Volume 47, No. 2, pp. 46-55, 2014.

[5] Gardner, E.M., "Which is the Correct Statistical Test to use?", British Journal of Oral and Maxillofacial Surgery, Volume 46, No. 1, pp. 38-41, 2008.

[6] Haller, K., " Mobile Testing", ACM SIGSOFT Software Engineering Notes, Volume 38, No. 6, pp. 1-8, 2013.

[7] Testing experience – Special Issue on “Mobile App Testing”, September, 2012.

[8] Muccini, H., Di Francesco, A., and Esposito, P., "Software Testing of Mobile Applications: Challenges and Future Research Directions", In Proceedings of 7th International Workshop on Automation of Software Test, pp. 29-35, 2012.

[9] Zhang, D., and Adipat, B., "Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications", International Journal of Human-Computer Interaction, Volume 18, No. 3, pp. 293-308, 2005.

[10] Yin, H., "Survey on Exploratory Testing", Master's Thesis, University of Tartu, Estonia, 2014.

[11] Whittaker, J.A., "Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design", Pearson Education Press, 2009.

[12] Madhushani, B.A.L., De Silva, P.H.A.M., Madushanka, W.A.L., Malalagama, M.G.T.H., and Manawadu, D., "Challenges in Mobile Application Testing: Sri Lankan Perspective", Compusoft, Volume 3, No. 10, pp. 185-199, 2014.

[13] "Pakistani Online Buyers Prefer Android Phones over Other Mobiles" [online], Available at: http://www.infopakistan.pk/2014/08/pakistani-online-buyers-prefer-android-phones-over-other-mobiles/.Accessed on Januray 17, 2016.

[14] Field. A., "Discovering Statistics using IBM SPSS Statistics", Sage Publications, 2013.

[15] Kanji, K.G., "100 Statistical Tests", Sage Publications, 2006.

[16] Pfahl, D., Yin, H., Mäntylä, M.V., and Münch, J., "How is Exploratory Testing Used? A State-of-the-Practice Survey", In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 5, 2014.

**Appendix-A:** https://figshare.com/articles/Appendix_A_pdf/
3119647