How our Data Science Department chooses what to research
By: Elad Cohen
Tags: Machine Learning, Management, Prioritization, Research, Data Science

How our Data Science Department chooses what to research

One of the biggest challenges in a Data Science organization is identifying and prioritizing the most promising directions to undertake. Here’s how we create our backlog and place our bets.

At Riskified, we have an extremely capable data science and research department, composed of over 30 data scientists and analysts. As a fast-growing startup we take a practical approach, measuring ourselves based on the actual business value delivered every quarter and how effectively we interpret data (more about it in my previous article). We use the Agile methodology and our own adaptation of Scrum to keep things moving at a high pace. Additionally, much of our team has grown in the fraud domain and we have a keen understanding of the business. This means that the majority of our quarterly planned work is driven by internal research initiatives with the minority coming from business stakeholders in other departments.

One of the biggest challenges in any quarterly plan is identifying and deciding on the most promising research directions to undertake. While we can’t say that we’ve got it all figured out, this blog will try to shed some light on the tradeoffs we consider and how we tackle this problem.

Our Structure

The data science department consists of several teams, half of which are product-oriented, and the other half centralized infrastructure-platform teams. The product-oriented teams help us commit resources to new budding products very early in their lifecycle. Each team directly impacts a specific product (with its set of KPIs) or contributes to a specific component of our flagship product, the chargeback guarantee solution. This split among the chargeback guarantee solution is by data science components so that one team owns the labeling, sampling and core algorithms, another team focuses on advanced featuring engineering, another on optimization of external data sources and so on. The most important aspect is that each team has KPIs to track their progress, and the contributions of these KPIs can be compared among the teams.

Planning process

At a high level, we follow the Design Thinking approach of divergent thinking (brainstorming) followed by convergent thinking (selecting options).

Divergent & Convergent Thinking

We start by creating a substantial list of potential research directions, which may include:

Relevant literature we’ve come across that generates interesting ideas

Conferences and meetups

Continuation/improvements of tasks currently being researched

Tasks derived from our annual plan (which is created through a similar, but much more rigorous and detailed process)

Infrastructure and other methods of reducing our technical debt

During these brainstorming sessions we try to come up with as many potential ideas as possible, building off of each other. Everyone comes prepared to these meetings, and discussing proposed concepts typically helps generate new ideas on-the-fly. Everyone is encouraged to offer ideas and research directions, regardless of their level of experience. This is usually a great place to witness diversity of thoughts and the advantage of having data scientists who come from different backgrounds.

At the end of this point, we add a bit more rigor and formalize the task — what the actual work is, and a first draft of the definition of done (DoD). We conclude this session with a vote from each team member for all the relevant tasks (this is done individually, with no discussion). After comparing all the votes, we look for any cases of large discrepancies between votes and debate these tasks — this is usually due to a misunderstanding between the team members around the research and/or the actual potential value. If necessary, we can hold another vote after these clarifications and finally come up with a prioritized list of research directions. While this intuitive Delphi method can work well, we also run these lists by management (group leads and department head) for their feedback. Looking back to a brainstorming session from May 2019 to direct work for Q3, we ended up completing 6 out of the 10 top-ranked research directions the team proposed during the ensuing months (testing new boosting algorithms was raised in that session).

While the original list of team ranked ideas can be a great start, there are a few additional considerations taken and heuristics used when evaluating the final priority of a research project.

Considerations in final prioritization

While the following list isn’t ranked, each point adds a different perspective on how we evaluate the priority of a project. We don’t maintain an explicit weighting function for each point and these implicit weights will change based on the business circumstances. While a lot of thought goes into the prioritization process, we aim to give the teams themselves as much autonomy as possible. The variation between the original list the team believes they should work on and the final version is typically very small — 10–20% of the work might get reprioritized due to the considerations below. It’s always better to have a good plan that gives the team a strong sense of ownership, than a great plan that feels dictated from above.

Historic projects in the same domain

At Riskified we’ve developed a taxonomy of research tracks. These include various families of features (our linking features, dynamic time-series features, behavioral and others), alongside data science domain tracks — sampling methodology, label cleaning, core algorithm changes, unsupervised algorithms. Before beginning a large research project focused on our sampling methodology, we’ll look at the track record of research in the same domain. While past work will never predict the potential success of upcoming tasks, seeing a historic high success rate can be a good indicator of more untapped potential.

At Riskified we’ve developed a taxonomy of research tracks. These include various families of features (our linking features, dynamic time-series features, behavioral and others), alongside data science domain tracks — sampling methodology, label cleaning, core algorithm changes, unsupervised algorithms. Before beginning a large research project focused on our sampling methodology, we’ll look at the track record of research in the same domain. While past work will never predict the potential success of upcoming tasks, seeing a historic high success rate can be a good indicator of more untapped potential. What works in the world

When reading about other industries that struggle with long maturation times (chargebacks can return after several weeks), dirty labels (we don’t know the true label of a declined order) and imbalanced data, we look to see what works well and what doesn’t. While we can’t be sure the resemblance is strong enough to our domain, seeing the same techniques utilized across multiple use cases with similar underlying problems is a great sign that it could be useful for us as well.

When reading about other industries that struggle with long maturation times (chargebacks can return after several weeks), dirty labels (we don’t know the true label of a declined order) and imbalanced data, we look to see what works well and what doesn’t. While we can’t be sure the resemblance is strong enough to our domain, seeing the same techniques utilized across multiple use cases with similar underlying problems is a great sign that it could be useful for us as well. Current state of affairs

Like any true agile organization, we strive to reduce our work in progress (WIP). This means we don’t want to be spread thin working on dozens of half-completed research projects that aren’t going to contribute value any time soon. We prefer to complete whatever is possible, reach a conclusion whether it is valuable or not and put it to bed before moving on to the next thing. Even if there’s a very interesting direction, we don’t want to start it if our current WIP is too high. While higher WIP seems innocuous, the value of WIP quickly degrades (are the results still relevant? Perhaps the population changed? Could the issue have been resolved due to other changes in the system?) and your people forget half-complete, not-fully-documented work. If data scientists start fresh tasks before completing the previous one, actual delivery can slow down dramatically.

Like any true agile organization, we strive to reduce our work in progress (WIP). This means we don’t want to be spread thin working on dozens of half-completed research projects that aren’t going to contribute value any time soon. We prefer to complete whatever is possible, reach a conclusion whether it is valuable or not and put it to bed before moving on to the next thing. Even if there’s a very interesting direction, we don’t want to start it if our current WIP is too high. While higher WIP seems innocuous, the value of WIP quickly degrades (are the results still relevant? Perhaps the population changed? Could the issue have been resolved due to other changes in the system?) and your people forget half-complete, not-fully-documented work. If data scientists start fresh tasks before completing the previous one, actual delivery can slow down dramatically. Implementation potential

We collaborate closely with our development team to make sure our research findings are implementable in our production systems. While some research won’t have any development implications (e.g. a change in our sampling methodology), making significant changes to our real-time decision mechanism can be a serious undertaking. We want to minimize the implementation time, as this is when value starts getting created. In some cases, e.g. when implementation is dependent on infrastructure, we can postpone the research to be closer to the development prerequisites.

We collaborate closely with our development team to make sure our research findings are implementable in our production systems. While some research won’t have any development implications (e.g. a change in our sampling methodology), making significant changes to our real-time decision mechanism can be a serious undertaking. We want to minimize the implementation time, as this is when value starts getting created. In some cases, e.g. when implementation is dependent on infrastructure, we can postpone the research to be closer to the development prerequisites. Plan readiness

Even when initiating exploratory work, it’s important to draw out the mental model of how the research is likely to go. You want to be able to picture a decision tree with different directions chosen depending on how the research will unfold. While it’s not practical to go 4–5 levels deep, it’s definitely useful to go the first 2–3 levels and make sure we can define the decision points and DoDs. We don’t want to greenlight projects that are overly open-ended or focus on exploring a domain without a possible end-game in mind.

Even when initiating exploratory work, it’s important to draw out the mental model of how the research is likely to go. You want to be able to picture a decision tree with different directions chosen depending on how the research will unfold. While it’s not practical to go 4–5 levels deep, it’s definitely useful to go the first 2–3 levels and make sure we can define the decision points and DoDs. We don’t want to greenlight projects that are overly open-ended or focus on exploring a domain without a possible end-game in mind. Value vs Risk

While we’d ideally want as many high-value, low-risk projects, this rarely happens. Typically, there’s a tradeoff between the potential value of a new idea and the likelihood it will actually succeed.

While you always want to take high-value, low-risk projects, you’ll typically choose a portfolio of research projects balancing safe bets with moonshots. | Diagram by Riskified

While we definitely want to take on as many projects that are low risk and high value (low hanging fruit), most projects will fall somewhere in the yellow quadrants (moonshots or safe bets). The best ideas usually encompass more radical changes and are higher-risk. The high value, high risk typically have a longer time horizon as well. While our main focus is reducing WIP and delivering value as soon as possible, this could create an incentive to shy away from moonshots. The potential concern is that you end up limiting your long term improvements. To make sure that we aren’t overly focused on short term projects we need to be sure of creating a steady pipeline of longer-term future improvements. This means sampling among both yellow areas, making sure we also choose a few research projects that are high potential, high risk, longer-term initiatives that are expected to bear fruit in more than six months (including implementation). Being overly minded on the short term projects can deliver an influx of value now, but hurt us (and our clients) in the long term. Maintaining this risk portfolio of short-term/low impact alongside long-term/high-risk/high-value projects is critical for long term success.

What don’t we consider?

Novelty

We’re totally driven by the value we deliver to our clients, and that means more accurate models. We don’t automatically adopt the newest technology or algorithm just because it’s new. We’d much rather be fast followers and let others demonstrate which technologies are here to stay. This also means we don’t focus on scientific publications at this point — while we have the utmost respect for people who publish, this would keep us away from delivering value to our clients.

We’re totally driven by the value we deliver to our clients, and that means more accurate models. We don’t automatically adopt the newest technology or algorithm just because it’s new. We’d much rather be fast followers and let others demonstrate which technologies are here to stay. This also means we don’t focus on scientific publications at this point — while we have the utmost respect for people who publish, this would keep us away from delivering value to our clients. Projects as the main driver of personal development

At Riskified we’ve initiated ongoing advanced data science tracks focused on our employees’ personal development which guarantees that they continue to learn new technical skills. We also encourage our data scientists to present their work to others in the department, externally at meetups and write tech blogs. Finally, we frequent professional conferences to stay up to date. We want our data scientists to grow and develop all the time, and not rely on their day to day projects as the main driver for their personal development (sometimes this works, but you don’t want to count on it). Given this approach, we are wary about choosing projects due to their impact on personal development. While some projects can be a great opportunity for data scientists to deepen their knowledge in a new field, we don’t initially consider this as a reason for a research project. We’ll revisit this decision if the final project list doesn’t offer adequate opportunities for personal development for certain individuals.

As with any fast-growing and dynamic company, there should be a lot more valuable projects to work on then you’ll be able to do. The prioritization process can be challenging and when done well, should generate healthy debate. Hopefully, this article sheds some light into how we prioritize our research and might help you prioritize yours.