(Re)Introducing the Data Marketplace
By: Thinkdata Works
Tags: External Data, Open Data, Data Platforms, Leveraged Trading, Datasource

Leveraging Data

According to Forbes, “external data is one of the biggest blind spots in executive decision making today.”

The future of business will be decided by companies that find ways to use more data: more of their own data, more public data, and more data generated by other businesses.

But when every external data source has its own intricacies, connecting to all this valuable data presents a problem many businesses are not equipped to solve.

As companies look for data generated outside the four walls of their organization, finding a scalable way to connect to these myriad sources becomes necessary.

One Single Source

Namara was born out of the idea that public data should be easier to find and access.

The original platform was focused primarily on open data — the data released by municipal, regional, and federal governments all over the world.

We built Namara to sit on top of these open data portals, standardize the data to a common format, and present it in one place. Searchable data, clean and simple.

It was, in our minds, a search engine for open data.

I’m feeling lucky?

Our Challenge

The problem, we realized, was that most people weren’t totally sure what open data was, or what was available. A search engine works because the people who use it already know what they’re looking for. In the case of a new resource like open data, the choice was paralyzing.

Furthermore, since we were pulling in data from so many sources a search for something simple like “schools” returned thousands of data sets from everywhere from Arvada County to the Yukon. It was clear that in order to facilitate the use of open data, we had to start making it not only easier to find, but easier to digest.

As we started building data products out of open data we became aware of the increasing availability of data that wasn’t necessarily “open” (i.e. not released by governments) but was obviously useful. After spending two years talking about the benefits of open data, we suddenly came to realize that it was only one piece of the puzzle.

A version of this graphic was first published by opensource.com

Search and Find

The next iteration of Namara reflected this realization. We continued to provide access to great open data, but also started piecing together data sets originating from alternative public sources.

This led to the idea of establishing a Data Marketplace — a place to search, yes, but also browse.

A big part of this was showing off data sets that we built both for clients and for ourselves. We standardized data from every government agency to provide the Treasury Board with an overarching view of government spending, and put the resulting data set — Canadian Procurement — into the Marketplace.

Every time we made a great data set we’d show it off in order to demonstrate the types of problems we were solving. After a while, the marketplace started looking less like a search engine and more like a place where you could connect to pre-built solutions.

But the world of public data is huge.

Although the Marketplace could show Namara users what we’d built, it wasn’t capable of showing off how much data is available, or what exactly went into the final data set. We could show products, but not components.

This is a problem because one of the great benefits of data is that a single data set can be used in so many different ways. Although oil is the most frequently cited analog for data, one of the major differences between the two resources (and there are a few) is that once oil is used, it’s used up. The same petroleum that powers your car can’t be used to heat your house or make plastic.

Data (a resource that actually creates more of itself as it’s being used) can do many different things for many different people, models, or applications simultaneously. By using the Marketplace to show off the products we’d already built, we were putting the data in a box and suggesting that this was its sole use.

As a company, we want to help more people do more with data (unleash the data scientist!). In order to do that better, we thought it was important to provide our users with the ingredients they can use, rather than show off the meals we’re making.

Browsing with Purpose: Re-designing the data marketplace

Roughly 10% of government data is published as open data. Add to the mix the availability of public (non-government) data, and the piece of the pie that open data represents becomes vanishingly small. It is with this in mind that we reconfigured the Data Marketplace to show off the sources of data that we’re connecting to.

The new Marketplace shows off dozens of choice sources and providers, as well as collections of the newest data, popular data sets, and specialized categories.

We’ve exposed some of the services we offer like Entity Resolution and Semantic Matching. We’ve also highlighted Featured Projects in a roster that will be constantly growing and updating as we uncover interesting and useful data.

Browse our sources and providers and find new data you might never have seen before. Or use our search function to get results from our massive repositories of refined and open data.

Also, don’t forget that we’re always interested in finding new sources of data. If you want to become a data provider or to recommend a data source, you can do it right in the Marketplace, or contact us here.