Data Transformation in PySpark. A step by step walkthrough of certain... | by Neel Iyer | Towards Data Science

Data is now growing faster than processing speeds. One of the many solutions to this problem is to parallelise our computing on large clusters. Enter PySpark.
However, PySpark requires you to think about data differently.
Instead of looking at a dataset row-wise. PySpark encourages you to look at it column-wise. This was a difficult transition for me at first. I’ll tell you the main tricks I learned so you don’t have to waste your time searching for the answers.
I’ll be using the Hazardous Air Pollutants dataset from Kaggle.
This Dataset is 8,097,069 rows.
df = spark.read.csv(‘epa_hap_daily_summary.csv’,inferSchema=True, header =True)df.show()
The first transformation we’ll do is a conditional if statement transformation. This is as follows: if a cell in our dataset contains a particular string we want to change the cell in another column.
Basically we want to go from this:
To this:
If local site name contains the word police then we set the is_police column to 1. Otherwise we set it to 0.
This kind of condition if statement is fairly easy to do in Pandas. We would use pd.np.where or df.apply. In the worst case scenario, we could even iterate through the rows. We can’t do any of that in Pyspark.
In Pyspark we can use the F.when statement or a UDF. This allows us to achieve the same result as above.
from pyspark.sql import functions as Fdf = df.withColumn('is_police',\     F.when(\     F.lower(\     F.col('local_site_name')).contains('police'),\     F.lit(1)).\                    otherwise(F.lit(0)))df.select('is_police', 'local_site_name').show()
Now suppose we want to extend what we’ve done above. This time if a cell contains any one of 3 strings then we change the corresponding cell in another column.
If any one of strings: 'Police', 'Fort' , 'Lab' are in the local_site_name column then we'll mark the corresponding cell as High Rating.
Therlike function combined with the F.when function we saw earlier allows to us to do just that.
parameter_list = ['Police', 'Fort' , 'Lab']df = df.withColumn('rating',\     F.when(\     F.col('local_site_name').rlike('|'.join(parameter_list)),\     F.lit('High Rating')).\     otherwise(F.lit('Low Rating')))df.select('rating', 'local_site_name').show()
F.when is actually useful for a lot of different things. In fact you can even do a chained F.when:
df = df.withColumn('rating', F.when(F.lower(F.col('local_site_name')).contains('police'), F.lit('High Rating'))\                              .when(F.lower(F.col('local_site_name')).contains('fort'), F.lit('High Rating'))\                              .when(F.lower(F.col('local_site_name')).contains('lab'), F.lit('High Rating'))\                              .otherwise(F.lit('Low Rating')))df.select('rating', 'local_site_name').show(
This achieves exactly the same thing we saw in the previous example. However, it’s more code to write and it’s more code to maintain.
I prefer the rlike method discussed above.
Whitespace can be really annoying. It really affects string matches and can cause unnecessary bugs in queries.
In my opinion it’s a good idea to remove whitespace as soon as possible.
F.trimallows us to do just that. It will remove all the whitespace for every row in the specified column.
df = df.withColumn('address', F.trim(F.col('address')))df.show()
Suppose we want to remove null rows on only one column. If we encounter NaN values in the pollutant_standard column drop that entire row.
This can accomplished fairly simply.
filtered_data = df.filter((F.col('pollutant_standard').isNotNull())) # filter out nullsfiltered_data.count()
The conditional OR parameter allows to remove rows where we event_type or site_num are NaN.
This is referred to as `|`.
filtered_data = df.filter((F.col('event_type').isNotNull()) | (F.col('site_num').isNotNull())) # filter out nullsfiltered_data.count()
df.na.drop allows us to remove rows where all our columns are NaN.
filtered_data = df.na.drop(how = 'all') # filter out nullsfiltered_data.show()
PySpark is still fairly a new language. Probably as a result of that there isn’t a lot of help on the internet. With something like Pandas or R there’s a wealth of information out there. With Spark that’s not the case at all.
So I hope these bits of code help someone out there. They certainly would’ve helped me and saved me a lot of time. The transformations I went through might seem small or trivial but there aren’t a lot of people talking about this stuff when it pertains to Spark. I hope this helps you in some way.
If I’ve made a mistake or you’d like reach out to me feel free to contact me on twitter.
Originally published at https://spiyer99.github.io on September 6, 2020.