Practical Approach to Promote System Test Automation in a Large-ScaleOrganization
Hideto Ogasawara∗1Takashi Nakano∗2∗1 Chiba Institute of Technology∗2 Toshiba CorporationIn addition to the development of new software for installation in various products, the creation of derivation software
utilizing existing software accumulated as development assets has been increasing. This approach, referred to as derivation
development, is eﬀective for the refurbishment of existing software. However, a large number of test cases need to be
examined in order to assure the quality of such software for a new product, making it diﬃcult to reduce testing worker
hours and shorten testing periods. The Toshiba Group has been implementing a broad array of measures to rectify thissituation through the development and introduction of system test automation techniques, as follows: (1) preparationof guidelines for the introduction and promotion of test automation, (2) development of test automation processes, (3)establishment of a test automation organization utilizing overseas resources, (4) maintenance of a uniﬁed management
environment for test speciﬁcations and test scripts, and (5) development of a technique to generate test scripts with easy
maintenance. We have conducted simulation experiments applying these techniques to a large-scale social infrastructure
system and conﬁrmed that the techniques make it possible to achieve a 30% reduction in the system testing costs ofsubsequent systems and a 40% reduction in initial introduction costs.
Keywords and Phrases: Test Automation Organization, Overseas Resources, Cost Reduction,
Test Automation Process

1. Introduction
In recent years, the functions and values of products have been provided by systems and digital information processing, and the development scale of software has dramatically increased. In the ﬁeld ofsoftware development, the importance of debugging and testing is increasing more to prevent problems
from occurring after product shipment. The time and worker-hours of the interface test and the systemtest in software development have reached about 40% of the whole development.
With the evolution and diversiﬁcation of information technology (IT) and the acceleration of thespeed of business and technological innovation, many test cases must be conducted in a short period oftime.
In software development of products, there are many derived derivatives which remodel existingsoftware rather than new development. Figure 1 shows the ratio of test worker-hours in derivation
development. Derivation development tends to decrease the cost of requirements analysis, design,implementation and test worker-hours from requirements deﬁnition to unit test as compared with newdevelopment. However, in the case of software development, changes to certain functions often aﬀect
diﬀerent functions. In addition, it is diﬃcult to accurately grasp the range of inﬂuence of change inadvance, so in many cases, quality is ensured by carrying out many tests including functions not directly
related to change in the testing process.
In software development of products, there are many derived derivatives which remodel existing
software rather than new development. Figure 1 shows the ratio of test worker-hours in derivation
development. Derivation development tends to decrease the cost of requirements analysis, design,implementation and test worker-hours from requirements deﬁnition to unit test as compared with newdevelopment. However, in the case of software development, changes to certain functions often aﬀect
diﬀerent functions. In addition, it is diﬃcult to accurately grasp the range of inﬂuence of change inadvance, so in many cases, quality is ensured by carrying out many tests including functions not directly
related to change in the testing process. For this reason, the time required for the test process could
not be greatly reduced, and the ratio of the test process to the total development tends to increase.
The smaller the scale of addition or change of speciﬁcation, the more testing worker hours of the test
process leads to cost increase of the whole of development.
The problems associated with an increase in testing worker hours are as follows:(1) Increase of worker hours by repeated test(2) Increase of backtracking worker hours due to omission of tests

Therefore, Toshiba group has developed a system test automation technology to verify whether theentire system meets the planned function in order to solve these problems. This paper describes theactivities of system test automation conducted by Toshiba group and its application eﬀect.
2. Problems and measures for system test automation
The process of system test automation is shown in Figure 2. In Step 1, system test execution is
automated by creating test scripts based on existing test speciﬁcations in order to solve the problem(1) mentioned in section 1. After that, it applies to multiple product series, and drastically reduces
testing worker hours of system test. In step 2, we develop a test speciﬁcation and test script automatic
generation technology in order to solve the problem (2) mentioned in section 1. By applying thesetechnologies, it is possible to develop products satisfying the requirements of QCD (Quality, Cos,Delivery).
On the other hand, Figure 3 shows the challenges faced by many organizations when introducing
and ﬁxing system test automation. Toshiba Group also introduced a test automation tool in multiple
development departments and tried it. However, the problems listed in Figure 3 became obvious, andthere were scattered circumstances in which system test automation was not continued.
Step 1 has Problem 1, 2, and 3. To summarize, ”It takes too much cost to introduce and maintainsystem test automation, so it will not be eﬀective”. Step 2 has Problem 4, and it became clear that”Because the current test speciﬁcation is insuﬃcient, it is not possible to eﬃciently and eﬀectively
create test scripts”. Taking these as an impediment factor in each step, we developed the measures inTable 1.
3. Eﬀorts to disseminate and deploy system test automation technology
We tried automating system test for software development of large-scale social infrastructure system.
First, we implemented the measures (1) to (4). Initiatives of measures (1) to (4) are shown below.
3.1 Measures (1): Overcoming introduction barriers
In order to overcome the introduction barriers concerning problem 1 and problem 2 in Figure 3, weselected tools, determined the scope of automation, and supported planning and execution managementof automation. Based on these experiences, we also created a guide to introduce and promote systemtest automation.
The system test automation introduction and promotion guide shows the steps of the strategy for
introducing system test automation in organizations and projects. In this guide, we provide reference
information such as what to do for each step, what should not be done.
Figure 4 shows the steps to develop system test automation strategy.
It is a point to clarify the purpose of automation (for example, improving the eﬃciency of systemtest) without aiming at system test automation itself, and to identify the object to be automatedwithout trying to automate everything. Therefore, merit and disadvantage of system test automation,
suitable or not suitable of system test automation are classiﬁed from three viewpoints, project features,software features, and test features.
Also, in order to prevent the activity from stopping when there was no person in charge of systemtest automation, we described that the system test automation activity should be incorporated into thedevelopment process of the organization. Furthermore, the necessity to clarify the roles of the designengineer, developer, test engineer, and test automation engineer, etc. was also stated.
3.2 Measures (2): Improvement of organization
Based on the description of the system test automation introduction and promotion guide, we developed an organization that automates the system test. Figure 5 shows the roles of software developers,
test developers, and support engineer. At the base of software development in Vietnam, we set up a test
center consisting of about twenty people. In order to increase the application of system test automation
within Toshiba Group, we integrated system test automation technology to this test center. Membersof the test center are assigned to the project as test automation developers and implement system test

automation. Utilizing the resources of Vietnam also led to cost savings in test script development andmaintenance. In addition, by clarifying the role sharing, we aimed to improve the quality of the system
testing by testing experts.
Toshiba’s Software Engineering Center provided support for designing the whole activities andholding regular meetings.
We informed the members of the test center the purpose and goal of the system test automation in
the development project and the future image of promoting the system test automation in the company.
And we shared that system test automation is an important activity for Toshiba.
We held two regular meetings of software developers and support engineers, test developers andsupport engineers every week. The main conﬁrmation / examination items at the regular meeting areas follows.
(1) Conﬁrmation of test script development status(2) Identiﬁcation of system test automatic target(3) Preempt risk for test script development and execution(4) Conﬁrmation of automated system test execution result(5) Conﬁrming the symptom that seems to be a problem
3.3 Measures (3): Improvement of the environment
Regarding problem 3 in Figure 3, the environment for system test automation was improved. Inorder to practice system test automation, it is necessary to manage stable system tests. If correspondence between the test speciﬁcation and the test script is not established, it becomes diﬃcult to manage
the system test result, and it takes time to modify the existing test script at the time of speciﬁcation
change or derivation development. In the worst case, reworking may occur. Toshiba Group widely uses
the test management tool TETRAPLUS which accumulates and manages test speciﬁcations and test
results. By using this tool, it is possible to conﬁrm and analyze the state of implementation of thetest in various views. In addition, a mechanism for reusing test assets (test speciﬁcations / results) isprovided.
In order to maintain the consistency of the test script and the test speciﬁcation, a system test
automation environment centered on the test management tool TETRAPLUS shown in Figure 6 wasdeveloped in cooperation with the test center.
Test scripts created with test automation tools on the market or free software and test speciﬁcations
registered in TETRAPLUS were drawn and the results of the tests automatically executed using thedeveloped linkage tool were saved in TETRAPLUS. As a result, test scripts that require modiﬁcationfrom the test speciﬁcation can be traced at the time of speciﬁcation change, etc., and uniﬁed management of the test script is realized. In addition, by registering the screen capture acquired at thetime of the test with the test automation tool on TETRAPLUS, evidence management of the systemtest result can be performed. Furthermore, with regard to the tests that it is diﬃcult to automaticallydecide whether or not to perform the system test, we devised so that the tester could see the screen
captured on TETRAPLUS and decide whether to accept the system test.
3.4 Measures (4): Develop test scripts that are easy to maintain
For eﬃciently executing the system test, Preprocessing, postprocessing, common processing, etc.
are cut out, and the structure of the data driven script which is easy to cope with the change of thesystem test environment and system test data was adopted.
4. Example of application eﬀect
We show examples of the eﬀect of system test automation on development of large-scale social infrastructure system.
We identiﬁed the scope of system test automation considering suitable or not suitable, reuse ofautomation, and judged that it is eﬀective if 50% of the existing test speciﬁcation could be automated,and we developed a test script. In this application example, 30% of system test automation is completed,and manual test cost can be reduced by almost 30%. This time it costed the initial introduction ofsystem test automation, but it is thought that the cost of the system test corresponding to this 30%
can be reduced after the next model.
Figure 7 shows the cost transition prediction of the system test. Utilizing the test center, theinitial introduction cost was reduced by 40% compared with the case where the system test automationis advanced only by its own department (the initial introduction cost in the case where system test
automation is realized only in its own department is a predicted value). As a result, it is possible tocarry out the test including the system test automation part at almost the same cost compared withthe previous model, and it can be seen that the initial introduction cost of automation can be almost
recovered by this development. In cost forecasting, by further promoting system test automation, thecost of system testing after the next model can be further reduced, and great cost reduction can beexpected from now on.
5. ConclusionWe were able to conﬁrm the cost reduction eﬀect by implementing eﬀorts to disseminate and deploy
system test automation against the problem of increase in testing worker hours. Automated systemtest utilizing Vietnam’s test center is spreading within Toshiba Group.
Currently, we are promoting measures (5) and (6) in Table 1, and we will promote system test
automation in terms of quality and promote the spread and deployment of system test automation in
the future.

References
Ogasawara, H., Sasaki, M., Nakano, T. and Aman, H. (2014). Application of the test select method
using mathematical programming model. EurpoSPI2014. 101-108.
Graham, D. (2014). Test Automation Problems - Survey results.