Lessons learned in Analytics Testing
By: Roberto Moschini
Tags: Analytics, Testing

During the past years, companies have unveiled an opportunity identifying data as a valuable asset to drive business growth. It has become more usual to get either analytics testing request or setting up specialized testing teams, but what does analytic testing truly means?

Applications, regardless of their platforms (Phones, Tablets, web browsers, Game Consoles, Smart TVs, casting Devices, 10 foot devices, etc), send anonymous information tracking users interactions, behaviors and platform specific properties. This powerful source of information aids the business understanding their users journeys and is a major tool not only for marketing campaign design but also for their progress evaluation.

The Basics

Analytics Test scope has to cover from the beacons fired by the Applications Under Test (That’s how the process of these variables gathering and transmission is commonly known) till they land on the different types of Analytics Servers.

The approach to run these tests is to set a proxy to capture the traffic originated by the applications and analyze the variables. Browsers have debugging tools where variables could be verified but not all the apps under test are web based. Charles Proxy is the most commonly used tool to capture and analyze these variables under test and Displays the data in a clear and well organized way under a working session that you can save and start and stop at your convenience. At first you will see all the network traffic going through your desktop and the connected device so probably one of the first actions will be to filter your view by removing the servers you don’t need to watch. That will keep the data manageable and small logs. Analytics data typically flows encrypted and in order to get it you will need to:

Ensure you install SSL certificates on the devices used for the test.

Coordinate with the development team the provisioning of a build allowing the required level of debugging.

Combining proxy tools with VPNs to access QA environments or mimicking international locations could be problematic and might require additional software, networking configuration and eventually additional hardware. If you manage to work with real time data pipeline debuggers they will allow you to check the data on line and is perhaps the most simplified and straightforward way to perform your validations.

E2E Approach

Validating the data being sent (When and What) is the main focus of the testing efforts but since there could be transformation rules at the end of the cycle at the Analytics Servers an E2E approach is recommended in order to verify how the data is presented to those using this information. This is challenging since we will not always get access to the organization Analytics Dashboards and even with appropriate access we will need support on how to identify and map every variable sent by the application. Relevant issues were discovered due to this approach where a transformation rule was excluding a platform from the usage of an application analysis.

Analytics Specialist Support

Testers have to usually jump into different business areas and move like a Subject Matter Expert, however when it comes to the major analytics Vendors tools, dashboards and Instrumentation strategies some specialized support is almost a must. You might succeed on the analytics variables validations but if you are willing to deliver more meaningful insights on the quality of the whole analytics approach the inclusion of a Digital Marketing Specialist to the team is more than desirable. They not only provide training and support to the team on the tools usage and dashboard creation but also bring insightful analysis and feedback on how the analytics strategy was implemented by the client.

Automation

There’s not a standard here but the most common approach is based on Selenium, plus a proxy like Browser Mob Proxy. Others work directly over the traffic at protocol level (enabling a wider scope of supported platforms and devices). In all cases you have to validate a large number of variables under several scenarios. Having Analytics Standards on your client side makes automation sustainable with a better ROI. In this section worth to mention Observepoint, a very common tool which provides journeys record and playback capabilities providing a way to have tests and execution results organized in a shared repository and fairly easy to use.

Analytics continues boosting business decisions and the pandemic accelerated the interest of companies in being more accurate tracking such information. Reliability and data loss avoidance will continue to be the driver for this test practice growth and as testers we should be prepared to jump into it.