DOI: 10.7910/dvn/i0ffjv
title: Survey of Income and Program Participation (SIPP)

description: &lt;h3 class="post-title entry-title" itemprop="name"&gt; analyze the survey of income and program participation (sipp) with r &lt;/h3&gt; if the census bureau's budget was gutted and only one complex sample survey survived, pray it's the survey of income and program participation (sipp). it's giant. it's rich with variables. it's monthly. it follows households over three, four, now five year panels. the congressional budget office uses it for their &lt;a href="http://www.cbo.gov/sites/default/files/cbofi
les/ftpdocs/87xx/doc8712/10-31-healthinsurmodel.pdf"&gt;health insurance simulation
&lt;/a&gt;. analysts read that sipp has person-month files, get scurred, and retreat to inferior options. the &lt;a href="http://usgsd.blogspot.com/search/label/american%20community%20survey%20%28acs%29"&gt;american community survey&lt;/a&gt; may be the mount everest of survey data, but sipp is most certainly the amazon. questions swing wild and free through the jungle canopy i mean &lt;a href="http://smpbff2.dsd.census.gov/pub/sipp/2008/l08puw1d.txt"&gt;core data dictionary&lt;/a&gt;. legend has it that there are still
species of &lt;a href="http://www.census.gov/sipp/top_mod/2008/topmod08.html"&gt;topical module variables&lt;/a&gt; that scientists like you have yet to analyze. &lt;a href="http://en.wikipedia.org/wiki/Fountain_of_Youth"&gt;ponce de león&lt;/a&gt; would've loved it here. ponce. what a name. what a guy.&lt;br /&gt; &lt;br /&gt; the sipp 2008 panel data started from a sample of 105,663 individuals in 42,030 households. once the sample gets drawn, the census bureau surveys one-fourth of the respondents every four months, over f
our or five years (panel durations vary). you absolutely must read and understand
&lt;a href="http://www.census.gov/sipp/usrguide/ch2_nov20.pdf"&gt;pdf pages 3, 4, and 5 of this document&lt;/a&gt; before starting any analysis (start at the header 'waves and rotation groups'). if you don't comprehend what's going on, try their &lt;a href="http://www.census.gov/sipp/tutorial/SIPP_Tutorial_Beta_version/SIPPtutorialHTML/S_group.htm"&gt;survey design tutorial&lt;/a&gt;. &lt;br /&gt; &lt;br /&gt; since sipp collects information from respondents regarding every month over the duration of the panel, you'll need to be
hyper-aware of whether you want your results to be point-in-time, annualized, or specific to some other period. the analysis scripts below provide examples of each. at every four-month interview point, every respondent answers
&lt;a href="http://smpbff2.dsd.census.gov/pub/sipp/2008/l08puw1d.txt"&gt;every core question&lt;/a&gt; for the previous four months. after that, &lt;a href="http://www.census.gov/sipp/top_mod/top_mods_chart.html"&gt;wave-specific addenda&lt;/a&gt; (called topical modules) get asked, but generally only regarding a single prior month. to repeat: core wave files contain four records per person, topical modules contain one. if you stacked every core wave, you would have one record per person per month for the duration o
f the panel. mmmassive. ~100,000 respondents x 12 months x ~4 years. have an analysis plan before you start writing code so you extract exactly what you need, nothing more. better yet, modify something of mine. cool? this new github repository contains eight, you read me, eight scripts:
&lt;br /&gt; &lt;br /&gt; &lt;b&gt;1996 panel - download and create database.R &lt;/b&gt;&lt;br /&gt; &lt;b&gt;2001 panel - download and create database.R &lt;/b&gt;&lt;br /&gt; &lt;b&gt;2004 panel - download and create database.R &lt;/b&gt;&lt;br /&gt; &lt;b&gt;2008 panel - download and create database.R&lt;/b&gt;&lt;br /&gt; &lt;ul&gt; &lt;li&gt;since some variables are &lt;a href="http://stat.ethz.ch/R-manual/R-patched/library/base/html/character.html"&gt;character strings&lt;/a&gt; in one file and &lt;a href="http://stat.ethz.ch/R-manual/R-patched/library/base/html/integer.html"&gt;integers&lt;/a&gt; in anoth
er, initiate &lt;a href="http://www.statmethods.net/management/userfunctions.html"&gt;an r function&lt;/a&gt; to harmonize &lt;a href="http://stat.ethz.ch/R-manual/R-devel/library/base/html/class.html"&gt;variable class&lt;/a&gt; inconsistencies in &lt;a href="http://smpbff2.dsd.census.gov/pub/sipp/2008/l08puw1.sas"&gt;the sas importation scripts&lt;/a&gt; &lt;/li&gt; &lt;li&gt;properly handle the parentheses seen in a few of &lt;a href="http://smpbff2.dsd.census.gov/pub/sipp/2008/lrw08_xx.sas"&gt;the sas importation scripts&lt;/a&gt;, because &lt;a href="h
ttp://blog.revolutionanalytics.com/2012/07/importing-public-data-with-sas-instructions-into-r.html"&gt;the SAScii package
&lt;/a&gt; currently does not&lt;/li&gt; &lt;li&gt;create an &lt;a href="http://cran.r-project.org/web/packages/RSQLite/RSQLite.pdf"&gt;rsqlite database&lt;/a&gt;, initiate &lt;a href="https://github.com/ajdamico/usgsd/blob/master/SQLite/read.SAScii.sqlite.R"&gt;a variant of the `read.SAScii` function&lt;/a&gt; that imports ascii data directly into a sql database (.db)&lt;/li&gt; &lt;li&gt;download each microdata file - weights, topical modules, everything - then read 'em into sql&lt;/li&gt; &lt;/ul&gt; &lt;br /&gt; &lt;b&gt;2008 panel - full year analysis examples.R&lt;/b&gt;&lt;
br /&gt; &lt;ul&gt; &lt;li&gt;define which waves and specific variables to pull into ram, based on the year chosen&lt;/li&gt; &lt;li&gt;&lt;a href="http://www.screenr.com/WgH8"&gt;loop through&lt;/a&gt; each of twelve months, constructing a single-year &lt;a href="http://www.sqlite.org/lang_createtable.html"&gt;temporary table&lt;/a&gt; inside the database&lt;/li&gt; &lt;li&gt;read that twelve-month file into working memory, then save it for faster loading later if you like&lt;/li&gt; &lt;li&gt;read the main and replicate weights columns into working memory too, &lt;a hre
f="http://stat.ethz.ch/R-manual/R-patched/library/base/html/merge.html"&gt;merge
&lt;/a&gt; everything&lt;/li&gt; &lt;li&gt;construct a few annualized and demographic columns using all twelve months' worth of information&lt;/li&gt; &lt;li&gt;construct a &lt;a href="http://faculty.washington.edu/tlumley/survey/html/svrepdesign.html"&gt;replicate-weighted complex sample design&lt;/a&gt; with a &lt;a href="http://www.amstat.org/sections/srms/proceedings/y2010/Files/307611_58945.pdf"&gt;fay's adjustment&lt;/a&gt; factor of one-half, again save it for faster loading later, only if you're so inclined&lt;/li&gt; &lt;li&gt;reproduce census-publish
ed statistics, not precisely (due to topcoding &lt;a href="http://www.census.gov/sipp/usrguide/chap4rev2009.pdf"&gt;described here on pdf page 19&lt;/a&gt;)&lt;/li&gt; &lt;/ul&gt; &lt;br /&gt; &lt;b&gt;2008 panel - point-in-time analysis examples.R&lt;/b&gt;&lt;br /&gt; &lt;ul&gt; &lt;li&gt;define which wave(s) and specific variables to pull into ram, based on the calendar month chosen&lt;/li&gt; &lt;li&gt;read that interview point (srefmon)- or calendar month (rhcalmn)-based file into working memory&lt;/li&gt; &lt;li&gt;read the topical module and replicate weights files into
working memory too, &lt;a href="http://stat.ethz.ch/R-manual/R-patched/library/base/html/merge.html"&gt;merge&lt;/a&gt; it like you mean it&lt;/li&gt; &lt;li&gt;construct a few new, exciting variables using both core and topical module questions&lt;/li&gt; &lt;li&gt;construct a &lt;a href="http://faculty.washington.edu/tlumley/survey/html/svrepdesign.html"&gt;replicate-weighted complex sample design&lt;/a&gt; with a &lt;a href="http://www.amstat.org/sections/srms/proceedings/y2010/Files/307611_58945.pdf"&gt;fay's adjustment&lt;/a&gt; factor of one-half&lt;/
li&gt; &lt;li&gt;reproduce census-published statistics, not exactly cuz the authors of this brief used the generalized variance formula (gvf) to calculate the margin of error - &lt;a href="https://www.census.gov/sipp/usrguide/chap7rev2008.pdf"&gt;see pdf page 4 for more detail&lt;/a&gt; - the friendly statisticians at census recommend using the replicate weights whenever possible. oh hayy, now it is.&lt;/li&gt; &lt;/ul&gt; &lt;br /&gt; &lt;b&gt;2008 panel - median value of household assets.R&lt;/b&gt;&lt;br /&gt; &lt;ul&gt; &lt;li&gt;define which wave(s) and spe
cific variables to pull into ram, based on the topical module chosen
&lt;/li&gt; &lt;li&gt;read the topical module and replicate weights files into working memory too, &lt;a href="http://stat.ethz.ch/R-manual/R-patched/library/base/html/merge.html"&gt;merge&lt;/a&gt; once again&lt;/li&gt; &lt;li&gt;construct a &lt;a href="http://faculty.washington.edu/tlumley/survey/html/svrepdesign.html"&gt;replicate-weighted complex sample design&lt;/a&gt; with a &lt;a href="http://www.amstat.org/sections/srms/proceedings/y2010/Files/307611_58945.pdf"&gt;fay's adjustment&lt;/a&gt; factor of one-half&lt;/li&gt; &lt;li&gt;reproduce census-published s
tatistics, not exactly due to topcoding (read more about topcoding by searching
&lt;a href="http://www.census.gov/sipp/usrguide/chap11rev2008.pdf"&gt;this&lt;/a&gt; and &lt;a href="http://www.census.gov/sipp/usrguide/ch10-rev2009.pdf"&gt;that&lt;/a&gt; user guide for, well, `topcoding`). huh. so topcoding affects asset statistics.&lt;/li&gt; &lt;/ul&gt; &lt;br /&gt; &lt;b&gt;replicate census poverty statistics.R&lt;/b&gt;&lt;br /&gt; &lt;ul&gt; &lt;li&gt;load a single wave of data &lt;/li&gt; &lt;li&gt;limit the table to the variables needed for an example analysis &lt;/li&gt; &lt;li&gt;construct the &lt;a href="http://faculty.washington.edu/tlumley/survey/html/svrepde
sign.html"&gt;complex sample survey object &lt;/a&gt;&lt;/li&gt; &lt;li&gt;print statistics and standard errors matching the &lt;a href="https://github.com/ajdamico/usgsd/blob/master/Survey%20of%20Income%20and%20Program%20Participation/SIPP%20PUF%20Poverty%20Statistics%20from%20Census.pdf?raw=true"&gt;target replication table&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;br /&gt; &lt;br /&gt; &lt;a href="https://github.com/ajdamico/usgsd/tree/master/Survey%20of%20Income%20and%20Program%20Participation"&gt;click here to view these eight scripts&lt;/a&gt;&lt;br /&gt; &lt;br /&gt; &lt;br /
&gt; &lt;br /&gt; for more detail about the survey of income and program participation (sipp), visit:&lt;br /&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="http://www.nap.edu/catalog.php?record_id=12715#toc"&gt;the reengineering of&lt;/a&gt; the survey of income and program participation, snore.&lt;/li&gt; &lt;li&gt;the survey of income and program participation &lt;a href="http://en.wikipedia.org/wiki/Survey_of_Income_and_Program_Participation"&gt;wikipedia entry&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;br /&gt; &lt;br /&gt; notes:&lt;br /&gt; &lt;br /&gt; sipp is right in the middle of an ultra-long pan
el, these scripts will update as new files are released.&lt;br /&gt; &lt;br /&gt; don't let the &lt;a href="http://www.census.gov/sipp/"&gt;deprecated-looking homepage&lt;/a&gt; dissuade you. the survey of income and program participation is happening now, red-hot. everything you need is available, albeit somewhat hidden. there's &lt;a href="http://www.census.gov/sipp/intro.html"&gt;a short introduction&lt;/a&gt;, &lt;a href="http://www.census.gov/sipp/DEWS/2008Schedule.pdf"&gt;the data release schedule&lt;/a&gt;, &lt;a href="http://thedatawe
b.rm.census.gov/ftp/sipp_ftp.html"&gt;an official ftp site&lt;/a&gt; - with &lt;a href="http://smpbff2.dsd.census.gov/pub/sipp/2008/l08puw1d.txt"&gt;codebooks&lt;/a&gt; - &lt;a href="http://www.census.gov/sipp/notes.html"&gt;advanced user notes&lt;/a&gt;, &lt;a href="http://www.census.gov/sipp/pubs.html"&gt;census publications based on sipp&lt;/a&gt; - don't miss &lt;a href="http://www.census.gov/sipp/tables/index.html"&gt;the table packages&lt;/a&gt; - aww cool even &lt;a href="http://www.census.gov/sipp/content.html"&gt;questionnaires&lt;/a&gt;. the &lt;a href="h
ttp://smpbff2.dsd.census.gov/pub/sipp/2008/l08puw1d.txt"&gt;core variable codebook
&lt;/a&gt; might not win any beauty pageants, but it'd be a wise use of time to slowly scroll through the first fifty variables. interview months take place after `srefmon == 4` and actual times of month and year can be determined with the `rhcalmn` + `rhcalyr` variables.&lt;br /&gt; &lt;br /&gt; perhaps more than any of the other data sets on this website, working with sipp will get more comfortable as you &lt;a href="http://www.pricewatch.com/system_memory/"&gt;increase your ram&lt;/a&gt;. so long as you manipulate t
hese files with &lt;a href="http://www.w3schools.com/sql/default.asp"&gt;sql commands&lt;/a&gt; inside &lt;a href="http://cran.r-project.org/web/packages/RSQLite/index.html"&gt;the sqlite database (.db)&lt;/a&gt; that my automated-download scripts create, you'll process these data line-by-line and therefore be untethered from any computer hardware limits. but the moment a &lt;a href="http://rss.acs.unt.edu/Rdoc/library/DBI/html/dbReadTable.html"&gt;dbReadTable&lt;/a&gt; or &lt;a href="http://svitsrv25.epfl.ch/R-doc/library/DBI/html/
dbSendQuery.html"&gt;dbGetQuery&lt;/a&gt; command pulls something into working memory, you'll begin gobbling up those precious four, eight, or sixteen gigabytes on your local computer. in practice, this simply requires that you to define the columns you need at the start, then limit what gets read-in to only those variables. you'll see it done in my scripts. if you don't copy that strategy -fair warning- you may hit &lt;a href="http://stat.ethz.ch/R-manual/R-patched/library/base/html/Memory-limits.html"&gt;
allocation errors&lt;/a&gt;. maybe keep the performance tab of your &lt;a href="https://www.google.com/search?q=windows%20task%20manager%20performance"&gt;windows task manager&lt;/a&gt; handy and &lt;a href="http://stat.ethz.ch/R-manual/R-patched/library/base/html/gc.html"&gt;take out the trash&lt;/a&gt;.&lt;br /&gt; &lt;br /&gt; &lt;br /&gt; confidential to sas, spss, stata, and sudaan users: &lt;a href="https://vimeo.com/52999628"&gt;watch this&lt;/a&gt;. time to transition to r. :D