Hypothesis Vetting: The Most Important Skill Every Successful Data Scientist Needs
By: Abderrahman Sobh
Tags: Machine Learning, Hypothesis Formation, Data Science, Kaggle, AI

The most successful Data Science starts with good hypothesis building. A well-thought hypothesis sets the direction and plan for a Data Science project. Accordingly, a hypothesis is the most important item for evaluating whether a Data Science project will be successful.

This skill is unfortunately often neglected or taught in a hand-wavy fashion, in favor of hands-on testing for feature significance and applying models on data in order to see if they are able to predict anything. While there is certainly always a need for feature engineering and model selection, doing so without true understanding of a problem can be dangerous and inefficient.

Through experience, I’ve derived a systematic way of approaching Data Science problems which guarantees both a relevant hypothesis and a strong signal as to whether a Data Science approach will be successful. In this article, I outline the steps used to do this: defining a data context, examining available data, and forming the hypothesis. I also take a look at a few completed competitions from Kaggle and run them through the process in order to provide real examples of this method in action.

Crafting the Perfect Hypothesis

1. Mapping the Data Context

Much like forming a Free Body Diagram in a physics problem or utilizing Object-Oriented Design in Computer Science, describing all valid entities in a given data context helps to map out the expected interplay between them. The goal of this step is to completely designate all the data that could possibly be collected about anything in the context i.e. a description of the perfect dataset. If all of this data were available, then the interactions between the components would be completely defined and heuristic formulas could be used to define every type of cause and effect.

2. Examining Overlap of Available Data to Data Context

Next, an observation is conducted to determine how much of the available data fits into the perfect dataset defined in the previous step. The more overlap found, the better a solution space for defining interactions between entities. While not a numeric metric, this observation gives a strong signal for intuition of whether the available data is appropriate and relevant enough. If there is a complete overlap, then a heuristic is probably a better solution than fitting a data model. If there is very little overlap, then even the best modeling techniques will be unable to consistently provide accurate predictions. Note that the strongest predictive signals will always be those that are directly related, thus indirect feature relation is given less emphasis in favor of the real thing. The intuition behind focusing mainly on strong signals is that collecting better data with strong signals will guarantee better performance and reliability. Conversely, predictions using weak signals are prime candidates to become obsolete once better data is available.

3. Forming the Hypothesis

Having completed the previous two steps, forming a hypothesis itself becomes trivial. Hypotheses are generally formed by combining a set of available features to predict the outcome of another feature that may be hard to collect in the future or whose value may be needed ahead of its outcome.

Now that the overlap between the data context and available data has been clearly defined, and assuming a reasonable amount of the available data is relevant/class-balanced/etc, a hypothesis can simply be stated as: “The available data can produce significant results predicting ____ in the given data context”. Note that the prediction label is left open-ended since it can potentially be filled in with many different things! This is because a hypothesis can be formed around almost any of the given features as long as it fits within the data context and there is enough overlap with the available data. Picking the specific feature to predict will depend upon what use cases are most beneficial.

Example Projects

Below, I’ve carefully selected a couple different Kaggle projects to analyze using the hypothesis forming technique. These examples illustrate some characteristic setups for hypothesis forming as well as each have a given hypothesis which can be evaluated in its respective data context.

Project 1: Predicting the Winner of a PubG Match (https://www.kaggle.com/c/pubg-finish-placement-prediction)

This project was selected because the data context is simplified, due to the nature that predictions are performed for a virtual world. Predictions of simulations are expected to behave much more consistently and operate along very clearly defined paths, as opposed to real-life systems which may have factors not easily observable. In other words, the data context can very confidently be exhaustively defined. In this video game, players control a single unit which can perform a limited set of actions. The data context has been mapped out below: