2019/06/19 test automation day report


9回目のTest automation day が 6/19-20 Netherlands Utrechtにて行われました

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/0bc1fac9-3dfe-6a1e-26da-2f27999b3a70.png" width=90%>

<B>Title : </B>Test Automation day
<B>Edition : </B>9
<B>Date :</B> June 19 and 20 2019
<B>Location :</B> Media Plaza, Jaarbeurs, Utrecht
<B>Outline :</B>
 In a world where everything needs to be continuous, has Test Automation also evolved into a continuous process: delivering software, testing, learning, creating value, etc.? How do you adjust to a continuous world? How do you keep up and deal with continuous change relative to Test Automation?


https://www.testautomationday.com/

そこで得たことをざっとまとめました
英語のプレゼンなので、ほとんど英語のままになってますw(日本語に変換するのが時間がかかるので)

# Balancing your test automation with exploratory testing

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/62658253-2691-ac77-7141-6820647672cc.png" width=80%>

Not automation testing vs exploratory testing!
lacking either is not good, not success testing!

#### How about automation testing and exploratory testing?

Automation improve speed up
 
- <B>create script based on known scenario which mean cannot guess testing</B>
- Regression testing
- Smoke testing
- Data driven testing
- Load testing ( same behavior
- Security testing  ( need compare explore testing
- CI and CD
- Anywhere , any time

Exploratory testing improve quality

- <B>find bug with your guess in unknown domain </B>, like 20 questions -> <A HREF="https://en.wikipedia.org/wiki/Twenty_Questions">20 questions </A>
- Functional testing
- Performance testing
- Load testing
- Stress testing (analyze data, change setting with feeling
- Security testing (hacker operation
- Usability testing (feeling
- Preparation for automated tests ( learn product , then decide which automated
- Any kind of software 
- <B>Review actives ( review specification document </B>

#### Summary
<B>Automation can find huge bug in huge area</B>
<B>Exploratory test can find small bug in focusing area</B>
Using both, we can speed up delivery and keep quality

# A Continuous Life in Test Automation

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/644debfd-17a0-110f-e871-28467fb039e6.png" width=80%>

彼女がこれまでの職において学んだことを順にお話ししていただきました。
その中で、テストにおいて変わったこと、変わらないこと、これからのことを挙げていました

### what has changed over the years?

- testing is a growing profession
 - none -> international qualifications for tester
 - tester -> test specialists , automation specialists, security specialists
- technical change 
 - mainframes -> mobile apps, cloud technology , IoT, ML/AI
 - homegrown utility -> commercial & free tools

### what hasn't changed?

- tester's greatest asset
- testing is still testing
- new technology, constant change
- managers don't understand testing lol
- <B>testing , automation seldom taught at university?</B>
- lots of people are new to testing/automation

### The future of testing ?

- <B>testing will always be needed</B>
- the essence of testing is what could go wrong?

### the future of automation ?

- automation will become more pervasive
- ML / AI will impact automation & testing
- people will do automation better
- <B>automation is necessary but not sufficient ( still chaos )</B>

# Will robots replace testers?

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/12a92c6b-e6dd-3636-d08c-0a59a0d4377c.png" width=80%>


- many testing tool and bots are coming and impact and influence to tester
- Testers will be replacing by tools ? Cannot say impossible
- New tools are going to emerge ,will support the thinking, exploring, recording, reporting, what testers do -> will replace tester!
- <B>a repetitive aspect , replaced by automation. But sophisticated analyses of data jobs , though process is not.</B>
- Tester need thinking tool, automation tool support only one “Applying” (Next Photo) , other 9 activities are not well supported
- Test automation tool is simple in principle , compared to the rest of the test process ..
- <B>Can we create tools to support tester thinking activities ?? like solving Wicked Problem.</B>
- Tester want to explore, but the need for control and documentation constrains them, tool will be led, requirement is Surveying metaphor
- <B>Compare the diagnosis of illnesses to testing?  firstly, dealing with people using many operations (listening heart , body scan etc) with a pleasant experience.  </B>
- Tester surveys are the source of data for Machine Learning bots , tools ,  like observations, ideas, concerns .
- on the other hands, more likely developers are affected by ML, unit testing, component development could be wholly automated

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/76c93773-9a56-7249-d614-c65ff7b9175c.png" width=80%>
reference report <A HREF="https://gerrardconsulting.com/mainsite/wp-content/uploads/2019/06/FutureOfToolsInTesting.pdf">The Future of Tools in Testing P5</A>


#  Automated functional acceptance testing of mobile apps using Robot Framework and Appium

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/4d19849f-d301-239b-1a43-3418ec69a289.png" width=80%>

### Robot Framework & Appium

Robot framework

- free, open source
- one framework for many applications
- keyword driven
- no program language skill

Appium

- free open source
- uses native tools to automate Android, iOS, windows OS
- tests native apps, hybrid apps, website

### Demo

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/3ec7f3f1-4fdd-32c3-4c44-2b7149e0160f.png" width=80%>

this framework is very easy scripting , easy understandable!


# Continuously Shipping Web Apps with Confidence 

Cypress tool : enable fast and reliable automated browser testing

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/b4603a05-4a0e-a772-ff1c-d22df55367fb.png" width=80%>

### whats Cypress ?

- A tool for reliably testing anything that run in web browser
- Free and open source MIT license
- Make testing an enjoyable experience , developer happiness
- Champion a testing process that actually works

### Benefit 

- Faster test runs
- Direct native access to the DOM & apps
- Better debuggability like dev tool
- Test commands are executed in a deterministic manner. Resulting in flake-free testing

### How do you run cypress in CI ?
- Headless mode
- Test run video recording out-of-the-box
- Optimize CI Usage with parallelization
- Test retires , reduce pipeline failures and identify true flaky tests ( upcoming )


# Mind the automation anti patterns

テスト自動化の様々な導入例、アンチパターン

### testing pyramid

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/eaf13473-676f-f847-6583-c4bf01899533.png" width=70%>

- test automation objective is to accelerate the feedback cycle
- mix of different test types
- bottom of the pyramid - large base and fast response
- higher up the pyramid - smaller set of tests , costly to develop, execute, maintain

### ice cream cone (ANTI PATTERN)

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/7617a49c-e6c9-05df-45a0-f761fe6b056b.png" width=70%>

- rare and non-existent test is automated
- automation is done through the GUI over the entire application landscape
- cannot achieve fast feedback

### cupcake pattern (ANTI PATTERN)

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/fbaef70e-5244-0fb0-c68d-ae8c1ae5c7f2.png" width=70%>

- everybody is involved in testing , but no communicating
- cause duplication testing

### hourglass (ANTI PATTERN)

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/56ce94ff-0a49-02ca-f011-2ba0ac420092.png" width=70%>

- a few test types are overrepresented
- some test level is too much, essential coverage is missing

### dual pyramid (ANTI PATTERN)

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/12248dce-13d1-f736-6612-c77a6e99006d.png" width=70%>

- different experts aren't collaborating (ex. developer and QA experts)
- cause a long feedback cycle

※ images from Xebia

# Reliable Software

### reliable software is different from functioning

- what's happen when air conditioner keep cool for a long time? something happens
- NASA satellite takes photo correctly in 13 years. but mistakenly didn't take photo in 2-3 days due to operational crush
- probability is low, but not be allowed issue　ex. <A HREF="https://www.cbsnews.com/news/toyota-unintended-acceleration-has-killed-89/">Toyota Unintended acceleration has killed 89</a>


### what's reliability?

<B> Probability of failure free software operation in specific environment for a specified period of time (IEEE 729)</B>

- something car works correctly
- but is it reliable? and how reliable is it?
- these reliable is different from environment, temperature, time, different users ...

### consider reliability with exercise 

locker system

- customer order something
- deliver stock items in security locker
- customer get items from security locker

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/7461cabf-c4ee-758c-5569-10e778675521.png" width=80%>

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/29c3264e-14d9-8881-d2d7-3896b17d0d66.png" width=80%>

#### 1. define customer profiles

- consider the locker system, meant for various settings
- think various different ways and locations to use this system
- is there a difference in users or usage?

#### 2. identify operation modes

- consider the residential profile
- two kinds pf users ( courier, customer )
- identify possible user actions

#### 3. identify possible failures

- consider the residential profile
- identify potential failures
- and assign to failure class
 - class1 : retry needed
 - class2 : traditional process used
 - class3 : unauthorized person can take package ( bad situation )

#### 4. define engineering processes

- think of the following areas
 - prevention : prevent reliability faults
 - detection : detect reliability faults
 - minimization of effects : design choices to minimize effects of faults
- identify typical activities

#### 5. time compression

- calculate necessary test time to demonstrate FIO (Failure Intensity Objective) of locker actions
- how can this be accelerated?

ex)
assumption : 
FIO cabinet target 1 failure in 5 * 10^5 locker actions
residential use , 225 apartments, 337.5 deliveries /wk , 56.3 deliveries /day
1 failure per 5.0E+05 locker actions => 1480 weeks = 28 years

ex)
accelerate :

- parallelize
- degradation : a little dangerous because not business use

### wrap up

reliabilityを考える上で以下ステップをふむ

#### 1st step define user domain reliability targets

- define customer profiles
- identify operation modes
- determine reliability requirements per operation mode
 
#### 2nd step derive software reliability targets

- define operational profile for user functions
- decompose software reliability targets
- decompose operational profile with probabilities per component

#### 3rd step define engineering processes

- process steps to prevent reliability faults
- process steps to detect reliability faults
- design choices to minimize effects of faults

#### 4th step measure software reliability growth

- design reliability tests
- run randomized tests
- visualize software reliability growth


# Finally

conferenceが行われた utrecht , Netherlands　ここはアムステルダムと異なりコジンマリした穏やかな都市でした
ユトレヒトはミッフィーの生まれ故郷で有名で、所々でミッフィーを見かけます。
アムステルダムへ行けば、ゴッホミュージアムもあり、そのコラボミッフィーをゲット!

<IMG SRC="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/267085/f343c04b-67f2-eeb1-2551-0ae9953dbf03.png" width=90%>