Do you stand by your data analysis?
By: Ron Soak
Tags: Data Analysis, Technology, Covid 19, Data, Reporting

Do you stand by your data analysis?

How we made sure our reports were watertight for COVID-19

Photo by Chris Liverani on Unsplash

When COVID-19 hit, many teams across Xero had to pivot their work to focus on immediate support for small businesses and their advisors (accountants and bookkeepers) around the world. A huge amount of change took place in a very short period of time, from government assistance packages to new ways of working.

To help our teams build features that supported these changes, we needed to provide more internal reporting. We also needed to share high-level insights to governments on the health of the small business sector.

Most data is not designed to be reported

A global pandemic is not the time to accidentally release inaccurate data analysis, especially when it’s also used by governments and industry analysts to measure the health of the small business sector. Our analysis was very much in demand, so we needed to make sure it was 100% accurate. That’s not to say we don’t normally check the accuracy of our work. We do. But data is complicated and often not structured with ease of reporting in mind.

In some cases, data that is customer friendly is not analysis friendly. For example, we allow customers to date invoices whenever they need to — today, next month, three years ago. This plays merry havoc on our ability to report on invoices, as invoices are always being entered in the past, causing historical numbers to change daily. Then there’s the fact that we don’t have one source of data — there are multiple streams of data for each product on our platform. And each stream is huge, often surpassing billions of rows of data. Like I said, it’s complicated.

We sadly don’t have a bank of pre-canned reports for every single eventuality. Instead, a problem arises and someone asks us to make a report with the data we have. We then combine data together to get an output. Sometimes the data isn’t designed to be combined, or needs to be combined in ways we couldn’t foresee when creating the data. For these reasons, it can be hard to make reports watertight. In some cases, the problem can’t be solved with the data we have.

At Xero, we’re very candid about the challenges we face when working with teams that ask us to do data analysis and reporting. This is especially true when we’re analysing something for the first time. We’re always doing our best and sometimes work with teams for years on the same problem to get it right. So how did we make sure we released accurate data analysis when we didn’t have years to get it right?

Ring-fencing our data analysis

We made two significant changes in our COVID-19 response. First, we ring-fenced all COVID-19 analysis. We have a lot of analysts at Xero who are quite independent in the work they do and the processes they follow. This gives our teams plenty of autonomy to do their best work, but wasn’t ideal in a pandemic. So we reached out to our analysts and let them know we were establishing a new process. This gave Xero confidence that we had oversight of all analysis as it was centrally managed.

Second, we looked at how we currently reviewed and released data analysis, and applied more rigour to it. For example, we made sure that only analysts familiar with the subject matter were creating the reports. It wasn’t the time for analysts to learn how to manipulate data in a new area. Likewise for those who reviewed the analysis. They also needed to be subject matter experts of that data. This gave us a lens to catch the nuances that would otherwise go undetected.

Then, in an effort to make sure the analysis didn’t succumb to effects of groupthink, each report (including its intent, methodology and results) was submitted to a cohort of peers to review in a process we referred to as the Data Use Submission. These peers included people from finance, customer, communications, governance, and legal teams. It also included other interested stakeholders and analysts.

The peer group reviewed everything from the individual logic we used, to the suitability of the output in comparison to the request. If you’ve ever been part of a critique, it was a lot like that, but over Slack and a shared Google doc. The results were then cross referenced with other supporting analysis, and in some scenarios counter analysis was performed. As a final check, the analysis was approved by the heads of governance, legal, finance, and communications before it was allowed to be released.

The amount of time and energy applied to every single analysis wouldn’t be considered appropriate in normal times — it made deadlines harder to meet as we were never sure just how much back and forth would be required during the submission process. But it was a necessary step in our COVID-19 response. One thing that made it easier was the fact that we could ask various people to pause what they were doing to help us, because it was so important.

Reflecting on what we learned

While it was a very manual and detailed process, it worked. Our new process forced everyone to look at what they were doing and ask: ‘If this goes out to the public tomorrow, and policy makers start making decisions based on this analysis, do I stand by it?’ It also gave us an opportunity to take some of the learnings and methods we used during this project and apply them to data analysis more broadly at Xero.

For example, some of the methods our peers used to test and question the way we create reports were highlighted in our retrospective as good techniques. The way we formatted our COVID-19 reports and the plain English language we used to explain our logic was also something we decided could be applied to many of our other processes. Working with such a diverse group of people also helped us understand each other’s worlds and as a result, relationships between teams became stronger.

Analysts who would never normally work together got to compare notes on how they solved similar problems, learning from each other’s experience. Teams who had traditionally just consumed the reports at the end of the process got to come along for the ride and learn what goes into creating a report. They also started to understand why reporting on data can be a real challenge. This was a huge win, because now they understand why we struggle or are slow to do things they perceive as easy.

The learnings also went the other way. I got to peek inside some of my stakeholders’ worlds to learn more about what they do and see how they responded to the demands of COVID-19. It was also interesting to see how our data analysis tied into what they produce — giving us a deeper insight into the value of data and how reports are actioned by stakeholders across the business. These were invaluable lessons that will do doubt help our work in the future.

Adapting to the new normal

I would say we did about one or two years’ worth of stakeholder engagement in about two months, which of course is paying dividends now. Although the COVID-19 crisis continues, we have a strong process and have now settled into a good rhythm. We interact with more teams on a regular basis, and there’s a shared understanding of what goes into ‘making the cake’ now they’ve seen it first hand. This has helped us find solutions faster, which benefits our teams and ultimately, our customers.

It will also help us as a data team going forward. Just like any company, we’ve planned to improve our data in the coming years so we can help customers get more value out of it. The difference is, now we know what’s missing and what couldn’t be achieved during COVID-19. That means we can focus on improving those areas, and we have some friends to help us when we’re ready to make those changes. As they say, a problem shared is a problem halved.

The processes we implemented to make sure Xero didn’t release inaccurate data analysis during a global pandemic wasn’t easy, but it helped so many people in so many ways. We got to make sure our data analysis and reporting was watertight, while bringing a whole lot of people at Xero along for the ride.