Practical Approach to Promote System Test Automation
in a Large-Scale Organization
Hideto Ogasawara*1 Takashi Nakano*2*1Chiba Institute of Technology *2TOSHIBA Corporation

削除:

In addition to the development of new software for installation in various products, the creation of derivation software utilizing existing

削除:

software accumulated as development assets has been increasing. This approach, referred to as derivation development, is effective for

削除:

削除:

the refurbishment of existing software. However, a large number of test cases need to be examined in order to assure the quality of such
software for a new product, making it difficult to reduce testing worker hours and shorten testing periods. The Toshiba Group has been
implementing a broad array of measures to rectify this situation through the development and introduction of system test automation
techniques, as follows: (1) preparation of guidelines for the introduction and promotion of test automation, (2) development of test
automation processes, (3) establishment of a test automation organization utilizing overseas resources, (4) maintenance of a unified
management environment for test specifications and test scripts, and (5) development of a technique to generate test scripts with easy
maintenance. We have conducted simulation experiments applying these techniques to a large-scale social infrastructure system andconfirmed that the techniques make it possible to achieve a 30% reduction in the system testing costs of subsequent systems and a 40%
reduction in initial introduction costs.
Keywords and Phrases: Test Automation Organization, Overseas Resources, Cost Reduction, Test Automation Process

1. Introduction
In recent years, the functions and values of products
have been provided by systems and digital information
processing, and the development scale of software has
dramatically increased. In the field of softwaredevelopment, the importance of debugging and testingis increasing more to prevent problems from occurring
after product shipment. The time and worker-hours of
the interface test and the system test in softwaredevelopment have reached about 40% of the wholedevelopment.

With the evolution and diversification ofinformation technology (IT) and the acceleration of thespeed of business and technological innovation, many
test cases must be conducted in a short period of time.
In software development of products, there aremany derived derivatives which remodel existing
software rather than new development. Figure 1 shows
the ratio of test worker-hours in derivation development.
Derivation development tends to decrease the cost ofrequirements analysis, design, implementation and test
worker-hours from requirements definition to unit test
as compared with new development. However, in theFigure 1 Comparison of ratios of testing cost in current and ideal derivation development

削除: pcase of software development, changes to certain
functions often affect different functions. In addition, it
is difficult to accurately grasp the range of influence ofchange in advance, so in many cases, quality is ensured
by carrying out many tests including functions notdirectly related to change in the testing process.
In software development of products, there aremany derived derivatives which remodel existing
software rather than new development. Figure 1 shows
the ratio of test worker-hours in derivation development.
Derivation development tends to decrease the cost ofrequirements analysis, design, implementation and test
worker-hours from requirements definition to unit test
as compared with new development. However, in thecase of software development, changes to certain
functions often affect different functions. In addition, itis difficult to accurately grasp the range of influence ofchange in advance, so in many cases, quality is ensured
by carrying out many tests including functions notdirectly related to change in the testing process. For thisreason, the time required for the test process could not
be greatly reduced, and the ratio of the test process tothe total development tends to increase. The smaller thescale of addition or change of specification, the more
testing worker hours of the test process leads to cost

increase of the whole of development.
The problems associated with an increase intesting worker hours are as follows:(1) Increase of worker hours by repeated test(2) Increase of backtracking worker hours due toomission of tests

削除:

Therefore, Toshiba group has developed asystem test automation technology to verify whether
the entire system meets the planned function in order tosolve these problems. This paper describes theactivities of system test automation conducted by
Toshiba group and its application effect.
2. Problems and measures for system test automation
The process of system test automation is shown inFigure 2. In Step 1, system test execution is automatedby creating test scripts based on existing test
specifications in order to solve the problem (1)mentioned in section 1. After that, it applies to multiple
product series, and drastically reduces testing worker
hours of system test. In step 2, we develop a test
specification and test script automatic generation

Figure 2 Two steps to promote system test automation

Figure 3 Problems in system test automation

削除:technology in order to solve the problem (2) mentioned
in section 1. By applying these technologies, it ispossible to develop products satisfying therequirements of QCD (Quality, Cos, Delivery).
On the other hand, Figure 3 shows the challenges
faced by many organizations when introducing andfixing system test automation. Toshiba Group also
introduced a test automation tool in multiple
development departments and tried it. However, theproblems listed in Figure 3 became obvious, and there
were scattered circumstances in which system test
automation was not continued.
Step 1 has Problem 1, 2, and 3. To summarize,"It takes too much cost to introduce and maintainsystem test automation, so it will not be effective". Step
2 has Problem 4, and it became clear that "Because thecurrent test specification is insufficient, it is notpossible to efficiently and effectively create test scripts".
Taking these as an impediment factor in each step, wedeveloped the measures in Table 1.
3. Efforts to disseminate and deploy system test
automation technology
We tried automating system test for softwaredevelopment of large-scale social infrastructure system.
First, we implemented the measures (1) to (4).
Initiatives of measures (1) to (4) are shown below.
3.1 Measures (1): Overcoming introduction barriers
In order to overcome the introduction barriers
concerning problem 1 and problem 2 in Figure 3, weselected tools, determined the scope of automation, andsupported planning and execution management ofautomation. Based on these experiences, we also
created a guide to introduce and promote system test
automation.

Table1 Measures to overcome hindering

Figure 4 Steps to develop test automation strategy

The system test automation introduction andpromotion guide shows the steps of the strategy for
introducing system test automation in organizations
and projects. In this guide, we provide reference
information such as what to do for each step, whatshould not be done.
Figure 4 shows the steps to develop system test
automation strategy.
It is a point to clarify the purpose of automation(for example, improving the efficiency of system test)
without aiming at system test automation itself, and toidentify the object to be automated without trying toautomate
everything. Therefore, merit
anddisadvantage of system test automation, suitable or notsuitable of system test automation are classified fromthree viewpoints, project features, software features,and test features.
Also, in order to prevent the activity fromstopping when there was no person in charge of systemtest automation, we described that the system test
automation activity should be incorporated into thedevelopment process of the organization. Furthermore,
the necessity to clarify the roles of the design engineer,developer, test engineer, and test automation engineer,etc. was also stated.

the system test automation in the company. And weshared that system test automation is an importantactivity for Toshiba.
We held two regular meetings of softwaredevelopers and support engineers, test developers andsupport engineers every week. The main confirmation /examination items at the regular meeting are as follows.

3.2 Measures (2): Improvement of organization
Based on the description of the system test
automation introduction and promotion guide, wedeveloped an organization that automates the systemtest. Figure 5 shows the roles of software developers,
test developers, and support engineer. At the base ofsoftware development in Vietnam, we set up a test
center consisting of about twenty people. In order toincrease the application of system test automation
within Toshiba Group, we integrated system test
automation technology to this test center. Members of
the test center are assigned to the project as test
automation developers and implement system test
automation. Utilizing the resources of Vietnam also led
to cost savings in test script development andmaintenance. In addition, by clarifying the role sharing,
we aimed to improve the quality of the system testingby testing experts.
Toshiba's Software Engineering Center provided
support for designing the whole activities and holding
regular meetings.
We informed the members of the test center thepurpose and goal of the system test automation in thedevelopment project and the future image of promoting

Figure 5 Roles of workers in charge of each section(1) Confirmation of test script development status(2) Identification of system test automatic target(3) Preempt risk for test script development andexecution(4) Confirmation of automated system test execution
result(5) Confirming the symptom that seems to be a problem

3.3 Measures (3): Improvement of the environment
Regarding problem 3 in Figure 3, theenvironment for system test automation was improved.
In order to practice system test automation, it isnecessary to manage stable system tests. If
correspondence between the test specification and thetest script is not established, it becomes difficult tomanage the system test result, and it takes time tomodify the existing test script at the time ofspecification change or derivation development. In theworst case, reworking may occur. Toshiba Groupwidely uses the test management tool TETRAPLUS
which accumulates and manages test specifications andtest results. By using this tool, it is possible to confirmand analyze the state of implementation of the test in
various views. In addition, a mechanism for reusing test
assets (test specifications / results) is provided.
In order to maintain the consistency of the test
script and the test specification, a system test
automation environment centered on the test
management tool TETRAPLUS shown in Figure 6 wasdeveloped in cooperation with the test center.

Test scripts created with test automation tools on
the market or free software and test specifications
registered in TETRAPLUS were drawn and the resultsof the tests automatically executed using the developedlinkage tool were saved in TETRAPLUS. As a result,test scripts that require modification from the test
specification can be traced at the time of specification
change, etc., and unified management of the test script
is realized. In addition, by registering the screen capture
acquired at the time of the test with the test automation
tool on TETRAPLUS, evidence management of thesystem test result can be performed. Furthermore, withregard to the tests that it is difficult to automaticallydecide whether or not to perform the system test, wedevised so that the tester could see the screen captured
on TETRAPLUS and decide whether to accept thesystem test.
3.4 Measures (4): Develop test scripts that are easy tomaintain
For efficiently executing the system test,Preprocessing, postprocessing, common processing, etc.
are cut out, and the structure of the data driven script
which is easy to cope with the change of the system test
environment and system test data was adopted.

4. Example of application effectWe show examples of the effect of system test
automation on development of large-scale social
infrastructure system.
We identified the scope of system test
automation considering suitable or not suitable, reuseof automation, and judged that it is effective if 50% of
the existing test specification could be automated, andwe developed a test script. In this application example,
30% of system test automation is completed, andmanual test cost can be reduced by almost 30%. This
time it costed the initial introduction of system test
automation, but it is thought that the cost of the systemtest corresponding to this 30% can be reduced after thenext model.
Figure 7 shows the cost transition prediction of
the system test. Utilizing the test center, the initialintroduction cost was reduced by 40% compared withthe case where the system test automation is advanced
only by its own department (the initial introduction costin the case where system test automation is realized
only in its own department is a predicted value). As aresult, it is possible to carry out the test including thesystem test automation part at almost the same costcompared with the previous model, and it can be seen

Figure 6 Established system test automation environment

削除:Figure 7 Changes in estimated system testing costs
that the initial introduction cost of automation can bealmost recovered by this development. In cost
forecasting, by further promoting system test
automation, the cost of system testing after the next
model can be further reduced, and great cost reduction
can be expected from now on.

Toshiba Group.
Currently, we are promoting measures (5) and(6) in Table 1, and we will promote system test
automation in terms of quality and promote the spreadand deployment of system test automation in the future.
References

5. Conclusion
We were able to confirm the cost reduction effect byimplementing efforts to disseminate and deploy
system test automation against the problem of increasein testing worker hours. Automated system test
utilizing Vietnam's test center is spreading within

Ogasawara, H., Sasaki, M., Nakano, T. and Aman, H.
(2014). Application of the test select method
using mathematical programming model, pp.101-10.8, EurpoSPI2014.
Graham, D. (2014). Test Automation Problems –Survey results, 2014.

削除: Hideto
削除: q
削除: Manami
削除: Takashi
削除: ，
削除: Hirohisa
削除:Dorothy
書式変更: インデント : ぶら下げインデント : 2.02 字,
最初の行 : -2.02 字削除: b
削除: