Why metadata matters
By: Jon Brock
Tags: Open Science, Data Science, Science, Blockchain, Metadata

Why metadata matters

Addressing science’s missing data problem by putting metadata on the blockchain

Photo by Tobias Fischer on Unsplash

When we think about the key features of blockchain — its decentralised, immutable, and universally accessible nature — it’s easy to see why people are excited about the potential for improving science. Blockchain technology is evolving rapidly, unlocking new opportunities for science and scientists.

At Frankl, we’re excited about the possibilities for the not too distant future. But our immediate concern is to provide pragmatic solutions that can be implemented today.

Our first step towards a blockchainified science involves metadata. By that we mean data about the data (when it was collected, what software was used, where it’s stored and so on) but not the actual data itself.

It sounds simple — and possibly quite boring). But putting metadata on the blockchain is ultimately a very powerful remedy to one of science’s biggest and oldest problems.

Where has all the data gone?

When we look at problems affecting science today, one of the most pressing issues is disappearing data.

Scientific data can go “absent without leave” for a number of different reasons:

Scientists don’t archive their data properly and they lose track of it, can’t make sense of it, or their hard-drive dies and they don’t have a back-up. This happens surprisingly (and embarrassingly) often. Scientists begin a study but abandon it before it is completed due to lack of funds, unpromising preliminary results, or other priorities. The data might be useful in combination with data from other studies, but it’s not publishable on its own. Scientists selectively publish data that supports a particular theory. Inconvenient data are quietly forgotten. Scientists try and publish data but are unsuccessful because the results aren’t considered interesting enough by the scientific journals. Knowing how difficult it will be to publish a null result, scientists prioritise writing up studies that gave them more publishable results.

The end result is what’s become known as the “file drawer problem”. The published scientific literature represents only a small and biased sample of the research that has actually been conducted. The rest is stuffed away at the back of the metaphorical filing cabinet.

There’s a lot of wasted effort here — data collected and then not used. But the bigger problem is the bias in what is published.

Publication bias in action

An example of this “publication bias” comes from my own research on autism.

In 2011, my colleagues and I published a study in which we gave a group of university students a simple “visual search” task that involved locating small objects in a complex display. We found that students with high levels of autism-like personality traits performed better on the task than those with fewer such traits.

It was an interesting finding because many studies have shown that people with an actual clinical diagnosis of autism also perform very well on such tasks. So our results were consistent with the idea that autism lies on a continuum that fades into the general population.

Another group in Western Australia reported similar findings around the same time, so we felt confident that this was a real effect. But a few years later, I was asked to peer review a study by researchers at Cambridge University. They had followed our methods exactly but with a much bigger sample — and found absolutely no effect.

At this point, I started asking other researchers in the field if they had looked at the same question. It turned out that several labs had done so, but hadn’t published their results because they didn’t find an effect.

Redundant research

This is just one example. But publication bias is a major problem throughout science. If we want to know whether an effect is real or not —whether that effect involves correlates of autistic traits or the efficacy of the latest cancer drug — we need to see the data in its messy totality. We don’t just want the data that tells a good story.

Scientists waste a huge amount of time and money on research that is effectively redundant — because the research has already been done but was considered unpublishable, or because it’s attempting to build on research that itself doesn’t stand up to scrutiny.

In a 2016 survey of over 1500 scientists, published in Nature, “Selective Reporting” was rated as the most important cause of irreproducible results.

The first step is that we need to know about all the data. And this is where blockchain can play a really important role.

Metadata on the blockchain

At Frankl, we’re building apps that facilitate open science by design. Data collected using Frankl apps will be archived on the fly (no more relying on dodgy hard-drives). Sharing data then becomes a simple exercise in changing the access privileges.

Where and in what form that data is archived will depend on a number of factors including the size of the data files, its private or public nature, as well as any ethical and legal considerations such as GDPR compliance. This will naturally vary from project to project and will evolve with technology.

But whatever happens to the actual data, the default setting will be for the metadata to be written to the blockchain. Whenever a Frankl app is used to collect data, a smart contract within the application will write a short message to the blockchain. This could include:

The time and date of data collection

The Frankl ID of the researcher

The identity and version number of the Frankl application used to collect the data

A pointer to the archived data (e.g., the URL of the repository)

The hash of the actual data

[The hash is a short character string derived from the data. You can’t recover the data from the hash. But if have the data, you can match it to the hash in the metadata and confirm that it hasn’t been altered.]

Why metadata matters

Writing metadata to the blockchain is a simple and easy first step. It depends only on existing technology (there’s no reliance on tech that is still in beta version). The metadata itself is small so the cost (i.e., ‘gas’) involved in writing to the blockchain is also small. And because there’s absolutely nothing that would allow anyone to identify the subjects of the research, there should be no ethical or privacy issues to contend with.

Yet even this small step is incredibly powerful. Because the blockchain is public and immutable, there’s no pretending that inconvenient data don’t exist. If scientists exclude data from their reported analyses, they have to justify it.

Public, immutable metadata helps prevent the actual data disappearing. It makes it easier to conduct meta-analyses that combine the results of different studies looking at the same effect (which can now include studies that haven’t been published and perhaps never will). It also provides a mechanism by which scientists can demonstrate their contribution to a collaborative project.

The beauty of metadata is not just its simplicity but its generality. We can employ the same structure to the metadata regardless of how large the related data file is, how it’s organised, where it’s stored, or even what scientific field it pertains to.

Metadata on the blockchain can be the thread that links all Frankl projects, the foundation on which all future developments build. It’s the first small but important step towards a blockchainified science — and away from science as normal.