Data Science for Social Good: Best Sources for Free Open Data
By: Ioana Spanache
Tags: Data, Open Data, Data Science, Machine Learning, Civictech

Data Science for Social Good: Best Sources for Free Open Data

Photo by Viktor Forgacs on Unsplash

Contents

Introduction What is Open Data? Open vs. Free vs. Online Data Where to find Open Data?

International organizations

United States

Europe

Latin America

Asia

Other Open Data sources (Google Public Data Explorer, Kaggle, FiveThirtyEight, UCI Machine Learning Repository etc.)

5. Conclusions

Introduction

Data Science has the power to bring great contributions to building the world we want to live in. And there are already numerous use cases which demonstrate how it can be leveraged for solving real-world problems.

Some examples of such cases can also be found in my previous article on this subject:

However, for doing so, we need data that is freely available for reusing and structured in a useful format. In this article, I am going through some of the most well-known and important portals that can be used in this regard.

What is open data?

‘Open data’ refers to data that are freely available without restrictions from copyright, patents or other mechanisms of control. (UNICEF Data)

In this context, it is not enough to just share data publicly in hard copy reports. For data to be considered fully open, it must follow certain principles that maximizes its utility:

to be structured using classifications accepted internationally (ISO-3166 for countries);

to use non-proprietary file formats (such as JSON or CSV);

to be available via standards-compliant communication interfaces (such as SDMX-JSON);

and have appropriate metadata describing it.

Open data is part of a larger set of movements, that includes also open-source software, open educational resources, open access, open science, open government and other.

More and more, certain types of data have started being considered a ‘public good’ which, when made available for use, reuse and free distribution can lead to better policy-making, better informed decisions, value creation and citizen-centric services. And this is how, Open Government Data philosophy and set of policies have also appeared.

Open government is a doctrine according to which citizens should have access to governmental documents and data for effective public oversight. By making governmental data open, public institutions show transparency and accountability in front of the citizens they are serving.

One amazing example I have encountered comes from Seoul, South Korea, where open data has become the norm and it is used for tackling real challenges the city and its citizens are facing. In Seoul, not only public institutions are using the data they are collecting, but also any business, non-profit organization or regular citizen can access them if they wish to build upon them or just check them for accountability reasons.

One of the goals of the City Hall is to provide open data to its citizens so that they can use them and build upon them. And by doing so, it has contributed to the creation of a new industry, in which many startups use the data provided for developing innovative solutions to some of the challenges faced within the city.

For more information regarding the example from South Korea and others alike, see the video below from The Economist:

Open vs. Free vs. Online Data

Open data is data without restrictions. Free data is data that is available without cost. Usually, open data is also free of charge. But when it comes to online data, not all of it can be used for free or without restrictions. In many cases it is copyrighted, being the propriety of its creators, and it requires permission or paying a fee.

Even when the data is not copyrighted, things are not perfectly clear. And we can think here of web scrapping data from LinkedIn. In 2019, the US Court of Appeals denied LinkedIn’s request to prevent the analytics company HiQ from scraping its data. Even so, LinkedIn does not appreciate anyone trying to scrape data from its platform, and warns against it in some articles.

Where to find Open Data?

Now, let's get to the meat of this article: where can one find open data; be it governmental or of other types. Below, I have covered sources of data provided by international organizations, sources specific to certain regions (US, Europe, Latin America, Asia), and other types of sources of global relevance.

International organizations

World Bank Open Data

Through this portal, the World Bank provides free and open access to a large palette of data regarding development in countries around the globe. And this comes as a result of their belief that by providing a broader access to their data, they increase transparency and accountability, as well as contribute to helping policy makers to make better informed decisions.

The users can navigate around the 4593 datasets either by country and regions or by indicators, organized around different sectors (agriculture, education, gender, infrastructure, environment, urban development etc.).

What is even more valuable about their search portal is the fact that it provides access to types of data such as time series, microdata (obtained from sample surveys, censuses and administrative systems), and geospatial data.

Moreover, if you wish to get a better impression regarding the type of information that can be extracted from their datasets, take a look at their 191 visualizations that cover topics such as the no. of people without access to electricity, the rise of global CO2 emissions, resource depletion, access to improved water sources etc.

OECD Data

The OECD Data portal provides access to 875 databases that can be searched according to the country of interest or topic (agriculture, development, economy, education, energy, environment, finance, government, health, innovation and technology, jobs, society).

One of the portal’s benefits is that it also provides data recorded over time, sometimes as early as 1959. One downside is that it covers mostly data related to countries which are part of the OECD. For example, Romania is not part of it.

And if you do not wish to download datasets yet, an just explore what they have in store, you can make your own queries on large databases in their data warehouse, OECD.Stat.

United Nations Data

The United Nation’s data portal has been created as a result of the belief that statistics should be considered a public good, which can serve for evidence-based policy and better informed decision-making.

The portal aims to provide free access to over 60 million data points organized in 32 large databases compiled by the UN, as same as by other international agencies in a single-entry point. Examples of source organizations are: Food and Agriculture Organization, World Health Organization, The World Bank, the OECD, International Monetary Fund etc.

The search engine allows users to look for information either based on the larger datasets, sources of data, or topics. Each such element has a drop-down menu which, in my opinion, allows for an easy user navigation.

Moreover, UN Data provides access to three specialized UNSD Databases, such as UNComtrade, Monthly Bulletin of Statistics Online and the well-known Sustainable Development Goals indicators, through separate individual portals. The UN Comtrade is a repository of official international trade statistics, relevant analytical tables and publications. MBS Online provide access to economic and social statistics regarding more than 200 countries and territories in the world. And it contains 55 tables with over 100 indicators on a variety of subjects, recorder for 80 years.

The United Nations Global SDG Database offers access to 460 data series that illustrate the progress registered towards achieving the Sustainable Development Goals. The search on the portal can be filtered either by goals and their specific targets and indicators, as same as by geographic areas (as it also includes country profiles) and years (2000 to 2019).

Some other features provided by the UN Data portal include access to popular statistical tables produced as part of the UN Statistical Yearbook and statistical profiles of countries (areas) and regions.

UNICEF DATA

The UNICEF DATA portal is for those wishing to work with data specifically about children and women. Their Data Warehouse includes datasets related to topics such as child mortality, child poverty, child protection and development, education, gender, maternal, child and newborn health, migration, nutrition, transition to work and other. And, again, data can be also filtered by country.

GHO data repository — World Health Organization

When it comes to data, WHO has a high coverage, as it works with 194 Member States from six regions. And through the Global Health Observatory, WHO provides access to more than 1000 indicators that it monitors that can be navigated either according to themes under SDG health and health-related targets, by category, or by country. Some examples of the types of data it provides are: road traffic injuries, noncommunicable diseases and mental health, mortality from environmental pollution, tobacco control, clean cities, Health Equity Monitor etc.