TITLE: Cleaning, Aggregating, and Filtering CMU Libraries Open Science and Data Collaborations Program Data
AUTHORS: Patrick Campbell, Huajin  Wang, Melanie  Gainey, Sarah  Young, Katie Behrman

[DESCRIPTION]
This document describes the process and tools used to clean, aggregate, and filter the data resources collected and maintained by University Libraries’ Open Science and Data Collaborations program at Carnegie Mellon University (CMU). This data cleaning protocol is used to support program evaluation and strategy development.

[STEPS]
SECTION: Designing Final Data Model
1. Each row in the final master dataset represents a unique user and columns represent attributes related to their identity and activity. The data is organized using the data model pictured below (Figure 1).  

SECTION: Cleaning
2. 
 
First, import all the component datasets (OSF, Workshops, LabArchives, KiltHub, Newsletter, Carpentries, dataCoLAB, and AIDR-OSS) into separate worksheets in a single Excel workbook. Refer to Data Collection Methods in the manuscript for more information about these datasets.

Then, for each worksheet, where necessary:

SECTION: Cleaning
3. Separate first and last names into separate columns called first_name and last_name, respectively.
SECTION: Cleaning
4. Use the Proper function to edit all name fields to use proper capitalization.
 

SECTION: Cleaning
5. Remove any periods after initials in name fields.
SECTION: Cleaning
6. Remove nonsensical characters.
SECTION: Cleaning
7. Reconcile equivalent emails (capitalization, etc.).
SECTION: Cleaning
8. Add institutional addresses for CMU students (institution = “Carnegie Mellon University”) with gmail addresses using CMU directory.
SECTION: Cleaning
9. Use conditional formatting in the first_name, last_name, and email fields to identify duplicate entries. Merge and delete duplicate records.
 

SECTION: Cleaning
10. Separate institution and department values into separate columns named institution and department, respectively (dataCoLAB table only).
SECTION: Cleaning
11. Use Find/Replace to replace all missing values (NA, Null, etc.) with an empty cell.
SECTION: Aggregation and merging
14. Create a new worksheet in the Excel workbook and name it “Master_Merged.”
SECTION: Aggregation and merging
15. Add the following fields in this order: ID, first_name, last_name, email, institution, department, user_kilthub, count_kilthub, user_la, type_of_account, no_of_notebooks, no_of_activities, user_nl, open_rate, click_rate, user_osf, no_of_projects_and_registrations, user_datacolab, user_carp, count_carp, user_workshop, count_workshop, user_aidr_oss, count_aidr_oss. 
 

SECTION: Aggregation and merging
16. Append the data from every component dataset into the new master worksheet “Master_Merged”) shifting the values over as necessary to match the field name at the top. 
 
 
SECTION: Aggregation and merging
17. Use conditional formatting to identify duplicate users using the first_name, last_name, and email fields across the different data sets. One by one, manually merge the data from all the rows for each user, deleting the duplicate records as you go.
 

SECTION: Aggregation and merging
18. Insert an empty column to the right of the institution column and name it institution2. In the top cell of the new column, use the IF function to test whether the user’s email contains either a CMU (@andrew.cmu.edu, @cmu.edu) or University of Pittsburgh (@pitt.edu) email extension. If the test returns TRUE, insert the name of the corresponding institution (“Carnegie Mellon University” for @andrew.cmu.edu and @cmu.edu extensions, “University of Pittsburgh” for @pitt.edu extensions). If the test returns FALSE, return the current value of the original institution column for that record.
SECTION: Filtering
20. 
 
Use the Filter tool to sort all records by their email address. Select all records with missing values for email and delete them. Turn the filter off.
SECTION: Harmonization Using the KiltHub Data Model
22. 
 

Filter the Institution column to only show CMU records.
SECTION: Harmonization Using the KiltHub Data Model
23. Copy the most recent complete version of the KiltHub data (see Data Collection Methods in the manuscript) into a separate sheet and delete all fields except the email and group (department) fields. Rename the sheet “KiltHub data model” to distinguish it from the KiltHub dataset that is already contained in your workbook.
SECTION: Harmonization Using the KiltHub Data Model
24. Back in the main worksheet (“Master_CMU_Pitt_only”), create three empty columns to the right of the department field. Name them department_kilt, department_merged, and department_cleaned, respectively.
SECTION: Harmonization Using the KiltHub Data Model
25. In the new department_kilt column, use the VLOOKUP function to search the email fields in the “KiltHub data model” sheet and return the department field (labeled group) for each matching email address from the master dataset (“Master_CMU_Pitt_only”). Use the Copy/Paste Values function to replace the formulas in the department_kilt column with just values.
SECTION: Harmonization Using the KiltHub Data Model
26. In the department_merged column, use the IF function to test if the value for department_kilt is null. If the value is not null, return that value from the department_kilt column. If the value is null, return the value of the original department field instead. Use the Copy/Paste Values function to replace the formulas in the department_merged column with just values.
SECTION: Harmonization Using the KiltHub Data Model
27. Copy the values from the department_merged column into the department_clean column. Sort the records in reverse alphabetical order (Z-to-A) by the values in the department_clean column.
SECTION: Harmonization Using the KiltHub Data Model
28. Compare the values from the department_cleaned column with the values from the department_kilt column. Identify corresponding department names between the two columns (e.g., “Biological Sciences” and “Department of Biological Sciences”) and use the Find/Replace function to replace the value in the department_clean column (e.g., “Biological Sciences”) to match the value in the department_kilt column (e.g., “Department of Biological Sciences”). 
 

SECTION: Harmonization Using the KiltHub Data Model
29. For the remaining non-matching records, refer to the CMU Academics webpage to determine the closest matching department from the department_kilt column and replace the value with that department name. 
 
If departments cannot be harmonized using this method, leave the values as they are.
SECTION: Harmonization Using the KiltHub Data Model
30. In the “Master_CMU_Pitt_only” worksheet, create a new column to the left of the first_name field and name it name_full. Use the concatenate function (=concatenate([first_name],” “,[last_name])) to combine the values from the first_name and last_name columns.
SECTION: Harmonization Using the KiltHub Data Model
31. In the “KiltHub data model” sheet, use the Find/Replace function to eliminate all periods (.) from the Author name field.
 

SECTION: Harmonization Using the KiltHub Data Model
32. Repeat steps 21 through 26 using the name_full field.
 

SECTION: Harmonization Using the KiltHub Data Model
33. Remove every department column except for the department_clean column. Rename the department_clean column as department.
SECTION: Harmonization Using the KiltHub Data Model
34. In the main sheet, use conditional formatting on the name_full field to identify any remaining duplicate entries. Merge the values from the duplicate records using the primary (institutional) email address as the primary record, copy/paste the secondary email address into the email_alias column, and delete the duplicate record.
SECTION: Harmonization Using the KiltHub Data Model
35. Use the Filter tool to sort records by department. For any record where the value in the department field is the name of an administrative office rather than an academic department or research center, replace that value with “Other”. Do the same thing for any record where the value in the department field is “Carnegie Mellon University” or “Faculty Other.”
SECTION: Harmonization Using the KiltHub Data Model
36. Insert a new column to the right of department and name it department_short. For any department name in the department column containing extraneous elements (e.g., “Department of,” “School of”, etc.), copy just the non-extraneous elements from into the new department_short column. For any department name in the department column still containing three or more words after removing all extraneous elements, copy just the acronym into the department_short column.
SECTION: Harmonization Using the KiltHub Data Model
37. Turn off all filters and correct any remaining obvious errors.
SECTION: Harmonization Using the KiltHub Data Model
38. Select “Master_CMU_Pitt_only” worksheet and save the file as "osdc_dept_final.csv".
SECTION: Cleaning
12. Add a column named user_<ToolName> that takes 1 or 0 values. Assign a 1 to all users to indicate that they are users of this tool or event. (0's will be added after the Aggregation and merging section for non-users of each tool or event.)
SECTION: Cleaning
13. Add another column and name it count_<ToolName>. (Note that there could be more than one count columns for some datasets, eg. LabArchives has both “count_notebooks” and “count_activities.”)
SECTION: Aggregation and merging
19. Highlight the contents of the institution2 column and use the Copy/Paste Values function to replace the contents of the column with the text values. Delete the original institution column and rename the institution2 column institution.
SECTION: Filtering
21. Use the Filter tool to sort all records by their institution. Select all records where institution does not equal either “Carnegie Mellon University” of “University of Pittsburgh” and delete them.