Package: unitids
get_unitids by Adam HearnA Python Module to Imputate IPEDS UnitID Numbers from Non-Matching Institution NamesHave you ever worked with institutional data from multiple sources? If so, one of them is likmayely IPEDS which of course involves the infamousunitidvariable. The secondary source, on the other hand, may only have the institution's name and nounitid. In this case, to join the datasets, you would need to merge on institution name and fill in the rest of the unitids manually to retrieve the IPEDS data.Anyone who has worked with IPEDS data would know that not all institution names perfectly line up across multiple sources. For example, Tulane University is named as Tulane University of Louisiana in the IPEDS universe. In this case of conflicting names, there would be an imperfect merge requiring you to manually enter Tulane's unitid number.In my background, I've run into this issue several times and has gotten to the point where it would be a better use of time to create a module to automate this step rather than filling out unitids manually. That said, I've developed the Python moduleunitids. I'm making this process open-source so other higher-ed researchers can benefit, too.This module, available in thepiplibrary, uses a cosine similarity text-analyis metric to merge partial or "non-matching" institution names with an IPEDS master file including all institutions in the IPEDS universe since 2004 and their unitid numbers.The process works by passing a DataFrame of institutions of which you want to get their unitids into theget_unitidsfunction. From there, the function populates a sparse matrix and generates a cosine-similarity metric for each insitution you passed and each institution in the IPEDS universe.It will then return two DataFrame objects: the first DataFrame will include your original data and the unitid numbers of the institutions in the dataset, along with a "match score" (displayed on a scale of 0-100).The second DataFrame includes information on the institutions that were not a perfect match, alongside their top-5 closes matches so you can make adjustments as necessary.ExampleTake, for example, the data present inthis article. The cleaned data is available on my Githubhere.Suppose we want to see the relationshpi between Forbes' Financial GPA and endowment, as reported to IPEDS. I've included a sample dataset of FY2018 endowment for all institutions in the IPEDS universehere.# Import necessary modules (no unitid, yet)importpandasaspdimportnumpyasnp# First we read in the Forbes data:url='https://raw.githubusercontent.com/ahearn15/get_unitids/master/example_dta.csv'forbes=pd.read_csv(url)forbes.head(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>CollegeStateFinancial GPAFinancial Grade0Stanford UniversityCA4.5A+1Massachusetts Institute of TechnologyMA4.5A+2University of Notre DameIN4.5A+3Princeton UniversityNJ4.5A+4University of PennsylvaniaPA4.5A+# Now we read in the IPEDS dataurl='https://raw.githubusercontent.com/ahearn15/get_unitids/master/ipeds_example.csv'ipeds=pd.read_csv(url).drop(columns='Unnamed: 0')ipeds.head(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>unitidinstitutionendowment0177834A T Still University of Health Sciences32420.01180203Aaniiih Nakoda College9212.02491464ABC Adult SchoolNaN3459523ABC Beauty AcademyNaN4485500ABCO TechnologyNaN# Now we merge togetherforbes=forbes.rename(columns={'College':'institution'})# need to rename for mergemerged=pd.merge(forbes,ipeds,on='institution',how="left")merged.head(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>institutionStateFinancial GPAFinancial Gradeunitidendowment0Stanford UniversityCA4.5A+243744.01496715.01Massachusetts Institute of TechnologyMA4.5A+166683.01454419.02University of Notre DameIN4.5A+152080.0837184.03Princeton UniversityNJ4.5A+186131.03053449.04University of PennsylvaniaPA4.5A+215062.0562092.0# How many did not merge?merged['unitid'].isna().sum()40If we merge our Forbes data with this 2018 list of IPEDS institutions, we get a successful merge rate of 95.7% (892 of 932 institutions). However, we still have 40 unitids we need to manually encode, taking up 40 lines of code and tedious trips to the IPEDS Data Center. What if we used the newget_unitidsfunction though?get_unitids# Installing the module!pipinstallunitids==0.0.92# Import required functionsimportnumpyasnpimportpandasaspdfromunitidsimportunitids#For viewing nonmatchespd.set_option('display.max_rows',100)Requirement already satisfied: unitids==0.0.92 in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (0.0.92)
Requirement already satisfied: numpy in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from unitids==0.0.92) (1.18.1)
Requirement already satisfied: pandas in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from unitids==0.0.92) (1.0.1)
Requirement already satisfied: nltk in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from unitids==0.0.92) (3.4.5)
Requirement already satisfied: textblob in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from unitids==0.0.92) (0.15.3)
Requirement already satisfied: pytz>=2017.2 in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from pandas->unitids==0.0.92) (2019.3)
Requirement already satisfied: python-dateutil>=2.6.1 in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from pandas->unitids==0.0.92) (2.8.1)
Requirement already satisfied: six in /Users/adamhearn/anaconda3/lib/python3.7/site-packages (from nltk->unitids==0.0.92) (1.14.0)Running the algorithmThe first argument we pass to the function,forbes, is our original dataset with the institutions of which we want to retreive unitids. The second argument,stateFlag, is an indicator of whether or not we have state abbreviations in our data. This makes the merge much faster and much cleaner, which I'll get to shortly.The funciton returns two DataFrames:mergedis our original dataset with the fancy new unitids. The second dataframe returned,nonmatches, allows us to investigate the institutions that did not perfectly merge and make adjustments as necessary. Now we're ready to call the function!Sidenote: For the algorithm to run error-free, the institution name variable must be listed the first column and the state variable (if available) must be in the second column.# Calling the functionforbes_unitids,nonmatches=unitids.get_unitids(forbes,stateFlag=True)#viewing the new dataforbes_unitids.head(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>institutionStateFinancial GPAFinancial Gradeunitidmatch0Stanford UniversityCA4.5A+243744.0100.01Massachusetts Institute of TechnologyMA4.5A+166683.0100.02University of Notre DameIN4.5A+152080.0100.03Princeton UniversityNJ4.5A+186131.0100.04University of PennsylvaniaPA4.5A+215062.0100.0# How many did not merge?forbes_unitids['unitid'].isna().sum()0100% of the institutions merged! We have significantly fewer unitids we need to fill in ourselves. We can investigate the institutions that were not perfect merges by viewing the second DataFrame,nonmatches:nonmatches<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>similarityunitidinstitutionmatchTop 5Baptist College of Health Sciences  (TN)1.0Baptist Memorial College of Health Sciences (TN)0.9258202196392.0Huntington College of Health Sciences (TN)0.8333334880683.0Huntington University of Health Sciences (TN)0.6666674880684.0American Baptist College (TN)0.6123722195055.0Fayetteville College of Cosmetology Arts and Sciences (TN)0.577350220163Calumet College of St. Joseph  (IN)1.0Calumet College of Saint Joseph (IN)0.8333331501722.0American College of Education (IN)0.547723449889Lincoln College of Technology (IN)0.5477231516613.0College of Court Reporting Inc (IN)0.500000150251PJ's College of Cosmetology (IN)0.500000152150Franklin & Marshall College  (PA)1.0Franklin and Marshall College (PA)0.8944272125772.0Harcum College (PA)0.577350212869Juniata College (PA)0.577350213251Rosemont College (PA)0.577350215691Wilson College (PA)0.577350217013Golden Gate University  (CA)1.0Golden Gate University-San Francisco (CA)0.8164971150832.0Ashford University (CA)0.577350154022Saybrook University (CA)0.577350123095Simpson University (CA)0.577350123457Stanbridge University (CA)0.577350446561Hobart and William Smith Colleges  (NY)1.0Hobart William Smith Colleges (NY)0.9128711916302.0The Sage Colleges (NY)0.4082481951283.0Berk Trade and Business School (NY)0.333333189219Bryant and Stratton College-Lackawanna (NY)0.333333374972Elim Bible Institute and College (NY)0.333333488305Jefferson (Philadelphia University + Thomas Jefferson University)  (PA)1.0Thomas Jefferson University (PA)0.9045342163662.0Philadelphia Biblical University-Langhorne (PA)0.5393602151143.0Bucknell University (PA)0.522233211291Lehigh University (PA)0.522233213543Widener University (PA)0.522233216852LeMoyne-Owen College  (TN)1.0Le Moyne-Owen College (TN)0.6708202206042.0Bethel College (TN)0.577350219718Maryville College (TN)0.577350220710Rhodes College (TN)0.577350221351Welch College (TN)0.577350220206Mount St Joseph University  (OH)1.0Mount Saint Joseph University (OH)0.8000002042002.0College of Mount St Joseph (OH)0.730297204200College of Mount St. Joseph (OH)0.7302972042003.0Mount Vernon Nazarene University (OH)0.600000204194University of Mount Union (OH)0.600000204185Paul Smith's College of Arts and Science  (NY)1.0Paul Smiths College of Arts and Science (NY)0.8249581943922.0SUNY College of Environmental Science and Forestry (NY)0.5892561961033.0St. Joseph's College of Nursing (NY)0.503953195191Vaughn College of Aeronautics and Technology (NY)0.5039531883404.0St. Peter's Hospital College of Nursing (NY)0.471405192961Saint John's University  (MN)1.0Saint Mary's University of Minnesota (MN)0.6761231748172.0Saint Johns University (MN)0.6708201747923.0Concordia University-Saint Paul (MN)0.600000173328Saint Cloud State University (MN)0.6000001747834.0Herzing University (MN)0.516398174154St. Ambrose University  (IA)1.0Saint Ambrose University (IA)0.7500001542352.0Dordt University (IA)0.577350153250Drake University (IA)0.577350153269Shiloh University (IA)0.577350480499Waldorf University (IA)0.577350154518St. Edward's University  (TX)1.0Saint Edward's University (TX)0.800000227845St Mary's University (TX)0.800000228149St. Mary's University (TX)0.8000002281492.0King's University (TX)0.670820439701St Marys University (TX)0.670820228149St. John Fisher College  (NY)1.0Saint John Fisher College (NY)0.8000001957202.0St Francis College (NY)0.670820195173St. Francis College (NY)0.6708201951733.0St Thomas Aquinas College (NY)0.600000195243St. Thomas Aquinas College (NY)0.600000195243St. Norbert College  (WI)1.0Saint Norbert College (WI)0.7500002397162.0Carroll College (WI)0.577350238458Edgewood College (WI)0.577350238661Lakeland College (WI)0.577350238980Shepherds College (WI)0.577350481137Tulane University  (LA)1.0Tulane University of Louisiana (LA)0.7745971607552.0Dillard University (LA)0.666667158802Herzing University (LA)0.6666674335363.0McNeese State University (LA)0.577350159717Nicholls State University (LA)0.577350159966For example, we can see that Franklin & Marshall College merged successfully with its official name in the IPEDS universe, "Franklin and Marshall College". Also, Tulane University merged successfully with "Tulane University of Louisiana", Hobart and William Smith Colleges merged with "Hobart William Smith Colleges", etc.Further, the algorithm accounts for institutions with changed names. For example, Dordt did not merge in the original dataset (since Forbes called it "Dordt College" and it's official IPEDS name is "Dordt University". The same issue is raised with Calvin College (now Calvin Univeristy). The IPEDS dictionary nested in the algorithm accounts for these historical name-changes. That is why there are only 15 non-perfect matches with this algorithm as opposed to 40 merging the "old fashioned way".However, this wasn't perfect: Saint John's University (MN) merged with Saint Mary's University of Minnesota instead, so we will need to correct that ourselves. We can find the correct unitid in thenonmatchesdataframe above. Still, one line of code compared to 40 is a big time-saver!# Replaces unitid for Saint John's University (MN) only# For Stata folks, same as replace unitid = 174792 if institution == "Saint John's University (MN)"forbes_unitids['untid']=np.where(forbes_unitids['institution']=="Saint John's University (MN)",174792,forbes_unitids['unitid'])Running this algorithm on this dataset as opposed to merging on institution-name gives us an accuracy of 99.9% (931 of 932 institutions), up from 95.7% earlier (892 of 932 institutions). It's a marginal improvement, but a big time-saver.To answer our original research question of how endowment impacts Forbes' "Financial GPA" measure, we can merge in our IPEDS data cleanly here.dta=pd.merge(forbes_unitids,ipeds,on='unitid',how='left')dta=dta.drop(columns="institution_y")# no need for duplicate columndta=dta.rename(columns={"institution_x":"institution"})# renamingdta.head(5)<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>institutionStateFinancial GPAFinancial Gradeunitidmatchuntidendowment0Stanford UniversityCA4.5A+243744.0100.0243744.01496715.01Massachusetts Institute of TechnologyMA4.5A+166683.0100.0166683.01454419.02University of Notre DameIN4.5A+152080.0100.0152080.0837184.03Princeton UniversityNJ4.5A+186131.0100.0186131.03053449.04University of PennsylvaniaPA4.5A+215062.0100.0215062.0562092.0%matplotlibinlineimportmatplotlib.pyplotaspltimportseabornassns#using logged endowment, as any sane econometrician wouldsns.scatterplot(x=np.log(dta["endowment"]),y=dta["Financial GPA"])<matplotlib.axes._subplots.AxesSubplot at 0x1a19878950>Seems like endowment plays a pretty significant role in Forbes' grading of Financial GPA!What if we have no state data?Note that the original merge went very well mostly due to us having access to state codes in our secondary dataset. Suppose our Forbes data did not have state codes, in which case the merge would have gone like this:forbes_unitids,nonmatches=unitids.get_unitids(forbes,stateFlag=False)Running the algorithm withstateFlag = Falsetakes significantly longer, considering we can no longer "throw out" institutions that do not match the same state. Instead, the algorithm must cross-check institutions across all states, not just the ones within states like it did whenstateFlagwas set toTrue.nonmatches<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>similarityunitidinstitutionmatchTop 5Baptist College of Health Sciences1.0Baptist Memorial College of Health Sciences (TN)0.9128712196392.0Australasian College of Health Sciences (OR)0.800000443599Bryan College of Health Sciences (NE)0.800000180878Jefferson College of Health Sciences (VA)0.800000231837Sentara College of Health Sciences (VA)0.800000232885Calumet College of St. Joseph1.0College of St Joseph (VT)0.8944272310772.0Calumet College of Saint Joseph (IN)0.800000150172College of Mount St Joseph (OH)0.800000204200College of Mount St. Joseph (OH)0.8000002042003.0St. Joseph's College of Nursing (NY)0.730297195191Franklin & Marshall College1.0Franklin and Marshall College (PA)0.8660252125772.0Franklin College (IN)0.816497150604Franklin College (TX)0.8164972257883.0Franklin Pierce College (NH)0.6666671827954.0Marshall Community and Technical College (WV)0.516398444954Golden Gate University1.0Golden Gate University-San Francisco (CA)0.7745971150832.0Chapman University-University College (CA)0.4714052620863.0Indiana University-Purdue University-Indianapolis (IN)0.436436151111University of Illinois University Administration (IL)0.436436149587University of Maryland-University College (MD)0.436436163204Hobart and William Smith Colleges1.0Hobart William Smith Colleges (NY)0.8944271916302.0College of William and Mary (VA)0.4000002316243.0Texas Barber Colleges and Hairstyling Schools (TX)0.3651484409894.0Richard Bland College of William and Mary (VA)0.338062233338State Board for Community and Junior Colleges (MS)0.338062247737Jefferson (Philadelphia University + Thomas Jefferson University)1.0Thomas Jefferson University (PA)0.9128712163662.0Thomas University (GA)0.6708201411673.0Saint Thomas University (FL)0.547723137476St Thomas University (FL)0.547723137476St. Thomas University (FL)0.547723137476LeMoyne-Owen College1.0Le Moyne-Owen College (TN)0.5773502206042.0Hussian College-Daymar College Clarksville (TN)0.436436368443Hussian College-Daymar College Columbus (OH)0.436436205559Hussian College-Daymar College Murfreesboro (TN)0.436436444255Hussian College-Daymar College Nashville (TN)0.436436220002Mount St Joseph University1.0Mount Saint Joseph University (OH)0.7500002042002.0College of Mount St Joseph (OH)0.670820204200College of Mount St. Joseph (OH)0.670820204200Mount St Mary's University (MD)0.670820163462Mount St. Mary's University (MD)0.670820163462Paul Smith's College of Arts and Science1.0Paul Smiths College of Arts and Science (NY)0.8017841943922.0University of Science and Arts of Oklahoma (OK)0.5892562077223.0Hope College of Arts and Sciences (FL)0.5773504883324.0Baker University College of Arts and Sciences (KS)0.534522154688Mayo Clinic College of Medicine and Science (MN)0.534522173957Saint John's University1.0Saint Edward's University (TX)0.750000227845Saint John's Seminary (MA)0.750000167677Saint Joseph's University (PA)0.750000215770Saint Martin's University (WA)0.750000236452Saint Peter's University (NJ)0.750000186432St. Ambrose University1.0St Bonaventure University (NY)0.666667195164St Lawrence University (NY)0.666667195216St. Andrews University (NC)0.666667199698St. Catherine University (MN)0.666667175005St. Thomas University (FL)0.666667137476St. Edward's University1.0Saint Edward's University (TX)0.750000227845St Mary's University (TX)0.750000228149St. Mary's University (TX)0.7500002281492.0Mount St Mary's University (MD)0.670820163462Mount St. Mary's University (MD)0.670820163462St. John Fisher College1.0Saint John Fisher College (NY)0.750000195720St John's College (MD)0.750000163976St John's College (NM)0.750000245652St. John's College (MD)0.750000163976St. John's College (NM)0.750000245652St. Norbert College1.0Saint Norbert College (WI)0.666667239716St Johns College (IL)0.666667148593St Petersburg College (FL)0.666667137078St Philips College (TX)0.666667227854St. Olaf College (MN)0.666667174844Tulane University1.0Tulane University of Louisiana (LA)0.7071071607552.0Chapman University-University College (CA)0.5773502620863.0Indiana University-Purdue University-Indianapolis (IN)0.534522151111University of Illinois University Administration (IL)0.534522149587University of Maryland-University College (MD)0.534522163204Union College1.0Union College (KY)1.000000157863Union College (NE)1.000000181738Union College (NY)1.0000001968662.0Columbia Union College (MD)0.816497162210Union County College (NJ)0.816497187198University of St. Thomas1.0University of St Thomas (MN)1.000000174914University of St Thomas (TX)1.000000227863University of St. Thomas (MN)1.0000001749142.0St Thomas University (FL)0.866025137476St. Thomas University (FL)0.866025137476Westminster College1.0Westminster College (MO)1.000000179946Westminster College (PA)1.000000216807Westminster College (UT)1.0000002308072.0Utah College of Massage Therapy-Westminster (CO)0.5773504691513.0Hussian College-Daymar College Nashville (TN)0.534522220002Wheaton College1.0Wheaton College (IL)1.000000149781Wheaton College (MA)1.0000001682812.0Hussian College-Daymar College Clarksville (TN)0.534522368443Hussian College-Daymar College Columbus (OH)0.534522205559Hussian College-Daymar College Murfreesboro (TN)0.534522444255There were still 913 perfect merges, but the accuracy on imputing the remaining 19 unitids is not as high: we only got 13 of these (accuracy of 68% within the nonperfect matches, 99.4% for the whole dataset).For example, Calumet College of St. Joseph merged with College of St. Joseph (VT) when it should have merged with Calumet College of Saint Joseph (IN). Plus, we must now account for institutions with the same name, such as Wheaton College (IL and MA). Still, not terrible, but it is alwasy a good idea to check for what unitids were imputed. Luckily this module makes it easy to do that.ConclusionI hope this can be a valuable tool for higher ed researchers out there. While this is currently only available in Python, I hope to get around to making it available on Stata and R soon. Please leave any feedback or bug reports on my GitHub repositoryhere.