Opening the Brazilian Census
By: André Coêlho
Tags: Python, Census, Ibge, Brazil, Microdata

Photo by Kaique Rocha from Pexels

The Brazilian Census is done every 10 years and the last census happened in 2010. Because of budget restrictions and Covid-19, the census will only take place next year, in 2022. The Brazilian Institute of Geography and Statistics (IBGE — Instituto Brasileiro de Geografia e Estatística) is responsible for the official collection of Censo. So, the Censo is used in a lot of researches and there is a lot of microdata on IBGE’s website. Therefore, it is important to know how to work with microdata from scratch.

First of all, the microdata can be downloaded, as well as the documentation. There are four different txt files, households, emigration, mortality and people (domicílios, emigração, mortalidade e pessoas). I’m using ‘Amostra_Domicilios_26.txt’, which is the data for the households for the state of Pernambuco, although any state can be downloaded.

The first thing to do is to import the pandas library. Then, the numbers inside the ‘width’ command below equals the numbers from the INT + DEC columns from the data dictionary file called ‘Layout_Amostra_Microdados.xls’ on the ‘DOMI’ tab, where it can be downloaded on the documentation link already mentioned.

To open the microdata ‘fwf’ is used, which stands for fixed-width format. It’s a file format where the data is arranged in columns, so there’s the need to know how long each column is, that’s why the ‘width’ command was chosen. ‘Amostra_Domicilios_26.txt’ is the name of the text file, ‘header=None’, which means there will be no header, and ‘dtype=str’ sets the data type to string.

import pandas as pd

width = [2,5,13,8,3+13,1,2,3,2,1,2,2,1,6,4+5,1,2,2+1,2,2+1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,

7,5+5,6+2,4+5,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]

ibgecenso2010domiciliospe = pd.read_fwf('Amostra_Domicilios_26.txt', widths=width, header=None, dtype=str)

Whenever opening a dataset, it is important to know how many rows and columns the data has respectively, using the ‘shape’ command.

ibgecenso2010domiciliospe.shape (253677, 76)

The ‘head’ command shows the first five rows, however, I prefer to use head(3) sometimes, showing only the first three rows.

ibgecenso2010domiciliospe.head(3)

As already known from the ‘shape’ output, the data has 76 columns, and it’s only showing the first 3 rows.

Nevertheless, the columns need to be renamed, and each variable name comes from the data dictionary ‘Layout_Microdados.xls’ file, which is inside the documentation. Even after renaming the columns look in the data dictionary file to see what each variable means.

ibgecenso2010domiciliospe.columns=['V0001','V0002','V0011','V0300','V0010','V1001','V1002','V1003','V1004','V1006','V4001','V4002',

'V0201','V2011','V2012','V0202','V0203','V6203','V0204','V6204','V0205','V0206','V0207','V0208','V0209','V0210','V0211','V0212',

'V0213','V0214','V0215','V0216','V0217','V0218','V0219','V0220','V0221','V0222','V0301','V0401','V0402','V0701','V6529','V6530',

'V6531','V6532','V6600','V6210','M0201','M2011','M0202','M0203','M0204','M0205','M0206','M0207','M0208','M0209','M0210','M0211',

'M0212','M0213','M0214','M0215','M0216','M0217','M0218','M0219','M0220','M0221','M0222','M0301','M0401','M0402','M0701','V1005']

Not all columns were shown, if someone wants to see every column, the command below can be used.

pd.set_option('display.max_columns',None)

You can see that the 3 dots between the columns disappeared.

Although there are many columns with detailed data, none of the variables in this whole microdata has the census sectors. There are many types of research with census sectors information, and it’s not here because the census sectors are on separate data.

Census Sectors

Photo by Caio from Pexels

The census sectors data are more detailed and there are smaller units than states and municipalities, like bairros and census sectors. Again, I am using the data for the state of Pernambuco, but the census sectors microdata can be downloaded for any state, and also the documentation can be downloaded.

The zip file contains 26 different CSV (and XLS) files, each file presents a lot of information about wage, race, sex, census sector, education, and much more. Within all the files, there are over 3200 variables!

Let’s open one of the files as an example:

Basico_PE = pd.read_csv('Basico_PE.csv')

Basico_PE.shape

(12379, 33)

This first file has 12379 rows and 33 columns. Using the ‘head’ command:

Basico_PE.head(10)

So, the first column has 15 digits, which is the ‘Cod_sector’, the census sector code. The table above is just from one of the 26 files, feel free to look at all the files with more than 3200 variables (columns), do not forget to look at the dictionary (‘BASE DE INFORMAÇÕES POR SETOR CENSITÁRIO Censo 2010 — Universo’) inside the documentation to see what each variable means.

Census Mersh

Photo by Pixabay from Pexels

Mesh of Enumeration Area or, simply Census Mesh, is mostly used for making maps and analyzing the census sectors. Once more, the census mersh for the state of Pernambuco is used. If anyone wishes to download another state, click on ‘Extensão’ (Extension), then on ‘SHP’, after that on ‘Malha de setores censitários’ (Mersh for Census Sectors’), then ‘2010’ and select the state you want. Let’s look at the file for Pernambuco by first importing the GeoPandas library:

import geopandas as gpd

pe_censusmersh = gpd.read_file('26SEE250GC_SIR.shp')

pe_censusmersh.shape

(12498, 14)

pe_censusmersh.head(3)

To make the map, the data above is merged with ‘Basico_PE’, the latter data was already mentioned before.

Basico_PE_merge_pe_censusmersh = Basico_PE.merge(pe_censusmersh, how='inner',left_on='Cod_setor',right_on='CD_GEOCODI', indicator=True)

The merge above used is ‘inner’ merge, to keep the common rows from both data, using the respective columns (Cod_setor, CD_GEOCODI) from each data (Basico_PE, pe_censusmersh). And ‘indicator=True’ shows the rows that were merged on both datasets if the merged row is from the left, right, or from both datasets.

Before making the map, there’s the need to convert the merged data frame into a GeoDataFrame, otherwise, the map won’t work.

from geopandas import GeoDataFrame

merged = GeoDataFrame(Basico_PE_merge_pe_censusmersh)

Also, import the library ‘matplotlib’ before making the map:

import matplotlib.pyplot as plt

f, ax = plt.subplots(1,figsize=(18, 18))

merged.plot(ax=ax, column='V005', legend=True, scheme = 'Quantiles',cmap='binary', edgecolor='k',

legend_kwds={'loc':'upper left', 'bbox_to_anchor':(1, 1), 'fmt':'{:.0f}'})

ax.set_axis_off()

ax.set_title('Monthly Average Nominal Income of Persons Responsible for Permanent Private Households', fontsize = 15)

plt.axis('equal')

plt.subplots_adjust(left=0.15, bottom=0.1, right=0.9, top=0.9, wspace=0.1, hspace=0.1)

plt.xlim(-41.5, -34.5)

plt.ylim(-12,-9.5)

fig.tight_layout()

plt.show()

As an example, the column ‘V005’ was used, and it represents the monthly average nominal income of persons responsible for permanent private households.