Demystifying Python for SAS users: how to perform common data management techniques in Python.
By: Eric Roberts
Tags: Data Management, Data Science, Python Pandas, Sas, Exploratory Data Analysis

Step 4: Recoding Variables

Recoding variables in SAS takes place exclusively in a data step but there is no equivalent function signaling data manipulation in Python. Instead you may code the manipulation but must be sure to save it to the dataset in one of several ways. Below I illustrate the difference between temporary and permanent transformations, show three ways to recode variables and end with the equivalent to SAS arrays.

Similar to SAS, if you want to transform numeric data you may write expressions using standard mathematical notation and pandas will broadcast the expression to every cell in the column. However, if you simply ask python to perform the calculation it will not permanently save the transformation to your dataset (Figure 5, line 14). One way to create a new variable is to simply instantiate a new column as shown in Figure 5 line 15. We tell python to create the variable ‘var4_converted’ in the ‘df’ dataset and set it equal to the transformation performed on ‘var4'.

Figure 5. Recoding a variable using broadcasting.

Figure 6 gives an example of an alternative notation you may run across using the loc indexer. The loc indexer is a way to refer to locations in a dataset using the labels of the columns or rows (as opposed to their numeric index with iloc). A colloquial interpretation of the below code is that within the dataset ‘df’ we use loc to locate all rows that satisfy the specified criteria and then in those rows set the column ‘var3_cat’ to the specified value. As the column ‘var3_cat’ did not previously exist it is created in line 17.

Figure 6. Categorizing a variable using the loc operator.

Figure 7 introduces three concepts: python missing values (NaN), apply, and lambda.

Figure 7. Setting system missing codes.

Notes —First, line 21 imports the numpy library and aliases it as ‘np’ by convention. It provides lots of numeric functionality; here I have imported it specifically as it handles missing values. NaN stands for ‘not a number’ and is analogous to the system missing ‘.’ in SAS. Second, in line 22, we are using the same formula as in figure 5 line 15: new variable = old variable + transformation. Third, we are using the apply function to perform the function wrapped in the the parentheses attached to the apply statement. The purpose of ‘apply’ is to perform the specified function to every cell in the column. Find more information on apply here. Fourth, we create a temporary function within parentheses attached to ‘apply’ by calling the key word lambda. Developers refer to it as the anonymous function as it only exists within that specific line of code. Next to the key word ‘lambda’ we specify the variable to use in our function. Here we use x but as with all loops this is just a place holder and we can use any letter. After the colon we write our function. One trouble I have had switching to Python is the different order of the logic in writing expressions. Whereas in SAS we write If X then Z, else Y. This expression reads Z if X else Y.

To end this post, I want to show one way you could recreate the functionality of a SAS array. Below is a simple array for a typical use case — recoding Don’t Know/Missing/Refused codes across many variables.

Figure 8. Sample SAS array.

Figure 9. For loop in python to recode multiple variables.

Similar to SAS we have to tell Python what variables to recode. Instead of defining the array in a data step, here we create a list of the variables to recode. This can be thought of as a global variable in that it exists in python’s memory and can be accessed again. We then use a for loop to perform the desired transformation on every variable in the list. The for loop performs the same function as the do loop and indexer (i in Figure 8). Note the difference between the result of Figure 9 and Figure 7. Figure 7 has created a new variable whereas the for loop in Figure 9 performs the transformation on the original variable.