How to boost Test Implementation Speed by 10x?
Testing As A Service

mesutdurukal

AGENDA
Preparation●●Implementation

Problem Definition
Requirement Analysis

Serve●●MaintenanceSupport

Chapter One

Preparation

Problem Statement● No standard across multiple teams/projects● Struggling with problems,
which were already solved somewhere else
● Duplication● Code quality● Reliability issues● No monitoringmyFramework.checkText(filterButton, '25 k')

Kayla

LeoSusanRobcy.get(button + ':visible' ) cy.get(button +cy.get(button +cy.get(button)':visible' ).click()':visible').should('be.visible').click().invoke('text').click().click().then((button)=>{.as('labelOnButton' );.should('contain',.should('have.text',expect(buttoncy.wait(2000);'25 k’);.text())'25 k’);
cy.get('@labelOnButton' ).then((actualLabel )=> {expect(actualLabel ).to.equal('25 k’)}).to.equal('25k’);})

Targets
●
●
●
●
●
●

Developing standards.
Avoid any antipatterns.
Remove duplication.
Provide solutions to common problems.
Provide an opportunity for monitoring
Dashboards, queries, filters, charts

Roadmap● Objectives & Key Results● Requirements definition● Prioritization

Objectives
and Key Metrics
QUALITY ASSURANCE

Improve Confidence
Quantity: Coverage
Quality: Escaped Bugs

Support Fast Delivery
Automation
Execution duration
Implementation
False alarms
Test fix time

Test coverage

90%

Number of critical incidents<3Automation coverage

100%

Total regression duration<10 mins

False failures<5%Average test fix time< 24 hours

Average test implementation time < 1 sprint

Challenges
●●●
●
●
●
●
●
●
●
●●●
●
●
●

Safari Automation
No infrastructure for automated regression testing
No proper Jira workflow for bugs
Impossibly testable cases
Unclear requirements / features
Infrastructure instabilities
Errors raised by the SUT
Cross domain redirection
Code changes to selectors / missing selectors
Proctor tests roll ups breaking automation
Test flakiness
Escaped bugs
Share tests between similar products
Use same approach for tests within a project
Keeping account profile data untainted
API testing framework compatibility

Requirements
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●

Testing BE and/or FE together should be supported
Rest API and GraphQL testing should be supported
API testing should support authenticated requests
FE testing modules should support Cookie operations
Including individual or a group of tests to the suite
Excluding tests from the executions should be possible.
Requests modification/interception should be supported
Response modification/interception should be supported
Test Input Management should be in place
Automatic bug reports after failures
Slack Notifications after executions
Evidence collection
Monitoring dashboards
There should be Hard fail and soft fail modesTests should be easily integrable to Gitlab/Github
Tests should be executable on multi-branch pipelines
Tests should be executable on different branches
Tests should be executable after commits/before merges

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

Cross domain testing should be supported
Clean up
Mocking should be supported
Various browsers should be supported
Mobile testing should be supported (native app)User manual should be generated
It should be easy to write new tests
Parallel execution should be supported
Test flakiness should be detected and reduced
Retry (line/whole test) strategy
No sensitive data should be revealed
Resources should be configurable
Static code analysis (linters, servers)
Dependencies should be auto-updated?
Accessibility testing components
Tests should have priorities
Async requests should be handled properly.

Chapter TwoImplementation

Benchmarking●●●●Community Research
Check other benchmarks
Self experience
Decision Criteria○ Speed○ Ease of coding○ Flexibility○ Documentation & Support○ Licensing / Cost
○ Feature Compatibility

Testcafe

Selenium

Cypress

Nightwatch

PlaywrightQuality Dimensions

Understandable
Transparent
Documentation
Troubleshooting

Flexible
Integrity
Scalable
Priority

Reliable
Coverage
Non Disruptive
Testability

Performance
Resource consumption
Multiple execution
Versioning

Flexible: Conﬁgurable, Cross Platforms

Integrity: Can integrate to CI platforms

Scalable/Portable: Parallelization, Desktop/Mobile

Goal Oriented: Priorities

Understandable

Transparent, Visible

Documentation: Read Me, Manuals

Troubleshooting, Evidences: SS, VideoReliable

Coverage

Nondisruptive

Testability: Mocks

Performance

Resource Consumption

Multiple Execution

VersioningChapter ThreeServiceSpeed Up Implementation●Define the locators○○●

Find the parent of an element
Find siblings and children

Implement test steps○○

Wait for responses, conditions
Parse Promises

page.locator(selector).click();vspage.click(page.locator(selector));page.locator(':text("label")');
vspage.getElementsByName("label");recentSearchesTitle().locator("xpath=..");
vspage.locator("article:has(" +recentSearchesTitle() + ")")

Eventually, looks like:Measure the effect!
Average:2 hours locators
3 hours
implementation

30 mins

Monitoring

TRAINING

Wrap Up
●
●
●
●
●
●

Observe Problems, Improvement AreasRequirements
Architecture
Implementation
Outcome
Monitoringmesutdurukal