select() vs selectExpr() in Spark | Towards Data Science

Column selection is definitely among the most commonly used operations performed over Spark DataFrames (and DataSets). Spark comes with two built-in methods that can be used for doing so, namely select() and selectExpr().
In today’s article we are going to discuss how to use both of them and also explain their main differences. Additionally, we will discuss when you should use one over the other.
Before discussing how to perform selection using either select() and selectExpr(), let’s create a sample DataFrame that we’ll use as a reference throughout this article
from pyspark.sql import SparkSessionspark_session = SparkSession.builder \    .master('local[1]') \    .appName('Example') \    .getOrCreate()df = spark_session.createDataFrame(    [       (1, 'Giorgos', 'Developer', 28),       (2, 'Andrew', 'Architect', 31),       (3, 'Maria', 'Manager', 31),       (4, 'Ben', 'Developer', 29),    ],    ['id', 'name', 'occupation', 'age'])df.show()+---+--------+----------+---+| id|    name|occupation|age|+---+--------+----------+---+|  1| Giorgos| Developer| 28||  2|  Andrew| Architect| 31||  3|   Maria|   Manager| 31||  4|     Ben| Developer| 29|+---+--------+----------+---+
pyspark.sql.DataFrame.select() is a transformation function that returns a new DataFrame with the desired columns as specified in the inputs. It accepts a single argument columns that can be a str, Column or list in case you want to select multiple columns. The method projects a set of expressions and will return a new Spark DataFrame.
The below expression will return a new DataFrame containing all the columns of the original DataFrame:
df.select('*').show()+---+--------+----------+---+| id|    name|occupation|age|+---+--------+----------+---+|  1| Giorgos| Developer| 28||  2|  Andrew| Architect| 31||  3|   Maria|   Manager| 31||  4|     Ben| Developer| 29|+---+--------+----------+---+
If you want to select only a subset of columns you can do so as shown below:
df.select('id', 'name').show()+---+--------+| id|    name|+---+--------+|  1| Giorgos||  2|  Andrew||  3|   Maria||  4|     Ben|+---+--------+
pyspark.sql.DataFrame.selectExpr() is similar to select() with the only difference being that it accepts SQL expressions (in string format) that will be executed. Again, this expression will return a new DataFrame out of the original based on the input provided.
Additionally, unlike select(), this method only accepts strings. For example, in order to multiply the column age times two along with the id andname columns, you need to execute the following statement:
df.selectExpr('id', 'name', 'age * 2').show()+---+--------+-------+| id|    name|age * 2|+---+--------+-------+|  1| Giorgos|     28||  2|  Andrew|     31||  3|   Maria|     31||  4|     Ben|     29|+---+--------+-------+
Therefore, select() method is useful when you simply need to select a subset of columns from a particular Spark DataFrame. On the other hand, selectExpr() comes in handy when you need to select particular columns while at the same time you also need to apply some sort of transformation over particular column(s).
In today’s article we discussed the difference between select() and selectExpr() methods in Spark. We showcased how both can be used to perform selection over Spark DataFrames and highlighted when to use one over the other.
You can read more articles about Spark below.