Technical introduction to the LIS databases: theharmonization process, database structure andaccess to data
Markus Jäntti1,21 Luxembourg Income Study 2 Swedish Institute for Social Research, Stockholm University

LIS Japan Workshop
13 January

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

1 / 39

Harmonization and database structure

LIS databases in general

General Framework for LIS

Deliverable: cross-national database that allows international
comparative research using micro-data
Scope: developed economies, to be extended with middle incomecountries

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

3 / 39

Harmonization and database structure

LIS databases in general

Harmonization and LIS

The harmonization process
The LIS databases
The LIS files + contentsMarkus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

4 / 39

Harmonization and database structure

LIS databases in general

The origins of the LIS surveys

LIS does not organize surveys
LIS collects household surveys that have:national coverage (all population groups, all geographic areas)
good quality income information (allows disposable income to bemeasured)
I labour market variables
II

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

5 / 39

Harmonization and database structure

LIS databases in general

The harmonization process

1

getting the data + documentation

2

file re-structuring (uniform units)

3

initialize standard set of output variables

4

mapping information into output variables

5

recode / standardize the values

6

apply standard missing values policy

7

annualize all income / expenditure

8

store final data in 3 formats (sas,spss,stata)Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

6 / 39

Harmonization and database structure

LIS databases in general

The harmonization processrecommendations:use best practices (Canberra report)
use international standard classifications (ISCO, ISCED, ISIC,ILO-definitions)provide extensive documentationuse contacts inside NSO + country experts

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

7 / 39

Harmonization and database structure

LIS databases in general

The harmonization processcomparability challenges:
surveys change over timecross-section versus panel (sample)
level of detail differs (HBS versus LFS)
net versus gross income
missing or imputed values

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 20118 / 39

Harmonization and database structure

LIS databases in general

All databases

LIS = income
LES = employment (discontinued)
LWS = wealthin total over 200 datapoints (majority LIS)

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 2011

9 / 39

Harmonization and database structure

LIS databases in general

All databases

unit of analyses per database
LIS = individuals + household
LES = individuals (adults)
LWS = households

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201110 / 39

Harmonization and database structure

The LIS databaseLIS database

the two dimensions are : country + timeIInearly 40 countries (OECD + Latin America)
nearly 4 decades span, grouped by wave

Number of countries increases with more recent waves over-time
analyses (be aware: not longitudinal!)

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201111 / 39

Harmonization and database structure

The LIS database

LIS database

LIS-specific anonimized identifiers :Datapoint identifier for country plus year (ccyy)
Household identifier to merge individuals and households
Person identifier (head, partner, etc.)

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201112 / 39

Harmonization and database structure

The LIS database

LIS database

Final output per datapoint
individuals : person file (ccyyp)household units : household file (ccyyh)
eventually plus a shadow file for eachMarkus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201113 / 39

Harmonization and database structure

The LIS database

LIS database

LIS Person filedemographic variableslabour market characteristics
income variables
identifiers
NOTE : every var ALWAYS exists, but not always filled ! Use Variable
Matrix to find out

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201114 / 39

Harmonization and database structure

The LIS database

LIS person file

Demographic Informationage, sex, relationship to the headmarital status / partnershipcitizenship, immigration status
general / vocational training
enrolled in education
disability status
see Quick Reference Manual

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201115 / 39

Harmonization and database structure

The LIS database

LIS person file

Labour market variableslabour force status
job characteristics, such as occupation, industry, sector
caregiving status , work experience

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201116 / 39

Harmonization and database structure

The LIS database

LIS person file

Income Variables that are meaningful at individual level
therefore, no such income as:
housing allowancechild allowance

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201117 / 39

Harmonization and database structure

The LIS database

LIS household filehousehold composition and characteristics
socio-demographic characteristics of head and spouseincome variables
expenditure variables

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201118 / 39

Harmonization and database structure

The LIS database

LIS household filehousehold characteristics
I

tenure of residence, region, urbanization

composition (derived from hhld-members)
number of persons, earners, elderly
number of children under 18*
I age of youngest child*
I number of persons under 14 (needed for “modified OECD”
equivalence scale)II* “child” = not head, nor spouse, never marriedMarkus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201119 / 39

Harmonization and database structure

The LIS database

LIS household filesocio-demographic characteristics of head and spouse
simply derived (copied) from person fileincludes age, marital status, lfs, education etc.

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201120 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Income Variables

large set of variables, covering all sources such as income fromlabour, property income, transfers, taxes
available over 3 levels of detail :main level, names starting with v such as unemployment benefits(v21)
I highly detailed sub variables
I aggregated summary income variables
I

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201121 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Income: subvariables

V8: Cash Property Income
interest and dividends v8s1
rental income v8s2
private savings plans v8s3
royalties v8s4
cash property income n.e.c. v8sr
available from wave IV onwards only

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201122 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Summary Income Variables

MI: Market Income
Wages and Salaries v1
Self-employment Income v4 + v5
Cash property Income v8
Occupational and other pensions v32 + v33

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201123 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Summary Income Variables

TRANSI: Total transfers
government transfers: social insurance benefits v16-v24
government transfers: social assistance (means-tested) v25-v26
private transfers v34-v35
DPI = net disposable household income

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201124 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Income variables
WARNINGnet versus gross datasets!
see overview on-line www.lisproject.org/techdoc/netdatasets.htm

DPI is gold standard
since this guarantees comparability
What is not included in DPI?
irregular lumpsums like lottery winnings, inheritance
imputed rent
non-cash income
REMINDER: all amounts in national currency

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201125 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Expenditure variables

Coverage: Total consumption expenditure as defined in theCOICOP classification
12 major groups, ranging from food, housing, medical
expenditures, etc, to miscellaneous
Total expenditure only filled when surveyed

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201126 / 39

Harmonization and database structureGroups of variables (LIS and LWS)

Important notes

Weighting
Missing values
HWEIGHT: Sample Weight
HWEIGHT: Sample Weight
Missing values

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201127 / 39

Data acces: LISSY, web tabulator and key figures

Data access: generalThree Pathways to access LIS Databases

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201129 / 39

Data acces: LISSY, web tabulator and key figures

Data access: generalThree pathways to access LIS Databases

The LISSY System
The Web Tabulator
The LIS Key Figures

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201130 / 39

Data acces: LISSY, web tabulator and key figures

Advanced data access: LISSY

The LISSY System

The primary means of access is a fully automated software
running 24/d 7/week designed specifically for LIS
SPSS, SAS or Stata batch programs submitted via a Job
Submission Interface (JSI) or an email software
LISSY automatically processes the jobs, generates results andreturns them to users on average within two minutes

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201131 / 39

Data acces: LISSY, web tabulator and key figures

Advanced data access: LISSY

Registration Process
http://www.lisproject.org/data-access/lissy.htm

Initial Registration process
Download, complete, sign and send back a pledge from the LIS
website, which states the rules governing the use of themicro-data
The userid and a password are strictly personal and must not
be shared with anyone
Researchers provide an email address when registerign. LISSY
returns output to this email address

Renewal Process
Access to the micro-data for a limited period of one year, renewableannually. Can be completed by annually updating information throughthe LIS website
http://www.lisproject.org/data-access/renewal.htm
Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201132 / 39

Data acces: LISSY, web tabulator and key figures

Advanced data access: LISSY

Structure of a job
Jobs submitted to LISSY require a few exceptions to the usual
program syntax
LISSY relies on a three-stage built-in alias to access datasets,consisting ofa statistical package specific heading (&,$ or ‘blank’)a two-digit ISO country(http://www.lisproject.org/techdoc/countryid.htm)
I an abbreviation used to identify the specific type of dataset

II$IT04h to access the LIS 2004 household Italian dataset (Stata)
LISSY rejects programs containing commands that allow users toprint or read individual records
We filter out jobs that risk breaching the rules on data
confidentiality. LISSY automatically puts the job in a security
review area to be manually reviewed by the staff
When syntax errors are detected, the language-specific error
message are display within the listing
Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201133 / 39

Data acces: LISSY, web tabulator and key figures

Advanced data access: LISSY

Submission via Email
Users submit job requests via outlook, thunderbird etc. topostbox@lisproject.org
Three requirements must be met in order for LISSY to properly
process jobs
All emails must be sent in ascii/plain text format. Users must ensurethat this option is enabled in the chosen email package.
I All job instructions must be written inside the body of the email andnot as an attachment.
I Each job must start with a specific four-line header at the very
beginning of the email body* user= your username* password = your password* package= your statistical package* project= LIS or LWS
I

Same exceptions to the user’s programming style are required
Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201134 / 39

Data acces: LISSY, web tabulator and key figures

The web tabulator

LIS Web Tabulator
http://www.lisproject.org/data-access/web-tabulator.htm

What is this?
An online table-making service enabling the design and generation ofcrossnational descriptive tables based on the underlying LIS datasets
without the need for programming

Registration required
Requires a username and password, see
http://www.lisproject.org/data-access/web-tabulator-registration.htm for the expedited registration
process

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201135 / 39

Data acces: LISSY, web tabulator and key figures

The web tabulator

LIS Web Tabulator
http://www.lisproject.org/data-access/web-tabulator.htm

Includes a restricted collection of datasets and variables
Datasets from LIS’ Wave V and Wave VI
Household-level data only
I Standardized indicators, including multiple indicators of realhousehold income, as well as demographic and labor market
variables
I Income variables expressed into 2005 international dollars
II

Two types of cross-national descriptive tables
Univariate Statistics and Cross-Classified Statistics tables
To-Do list to guide designing tables
I Possibility to export aggregated results in EXCEL or in ASCII
formatsII

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201136 / 39

Data acces: LISSY, web tabulator and key figures

Key figures

LIS Key Figures
http://www.lisproject.org/key-figures/key-figures.htm

Two distinct sets of LIS Key Figures
LIS Inequality and Poverty Key Figures comprise national-level
inequality and poverty indicators such as Gini Coefficients,
Atkinson coefficients, etc (all waves)
LIS Gender Key Figures that include national-level indicators
highlighting women’s economic outcomes and gender inequality inpoverty and employment (Wave V and soon Wave VI)

LIS Key Figures exist in two formats directly accessible
Each set of Key Figures is included in a downloadable Excel
workbook
The Search Engines allow the creation of subsets of the two setsof LIS Key Figures to narrow the scope of the analysis.
Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201137 / 39

Data acces: LISSY, web tabulator and key figures

Key figures

LIS Key Figures

Programming routines and information are publicly available
All methodological decisions are transparent
Alter the programs to fit the needs of individual research projects

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201138 / 39

Data acces: LISSY, web tabulator and key figures

Finally

Last Minutes . . .

If you have any problems contact the LIS user support
usersupport@lisproject.org
For users who may be unfamiliar with batch coding US00 (UnitedStates, 2000) and IT00 (Italy 2000) (sub-sample) files aredownloadable(http://www.lisproject.org/self-teaching.htm)
Key figures programs (http://www.lisproject.org/
key-figures/key-figures-programs.htm)
If you have any problems contact the LIS user support
usersupport@lisproject.org

Markus Jäntti (LIS/SOFI)

LIS/TechnicalJanuary 201139 / 39