Question: I'm working with PySpark and have a challenging scenario where I need to dynamically retrieve the value of a field specified in another field of the same DataFrame. I then need to compare this dynamically retrieved value with a fixed value. Here’s the context: I have a DataFrame df_exploded with the following schema: root |-- \_id: string (nullable = true) |-- name: string (nullable = true) |-- precondition: struct (nullable = true) | |-- field: string (nullable = true) | |-- matchingType: string (nullable = true) | |-- matchingValue: string (nullable = true) |-- businessTransaction: struct (nullable = true) | |-- someField1: string (nullable = true) | |-- someField2: string (nullable = true) | |-- someField3: string (nullable = true)  The precondition field is a struct containing:  field: the name of the field in businessTransaction whose value I need to compare. matchingType: the type of comparison (e.g., equals, greater than). matchingValue: the value to compare against.  Objective: For each row, I need to dynamically retrieve the value from the field specified in precondition.field (which can vary across rows) within businessTransaction and compare it with precondition.matchingValue. Question: How can I dynamically retrieve and compare the value from a field specified in another field of the same DataFrame in PySpark? Is there a way to use expr or another function to evaluate the column path stored in dynamic_field_path to get the actual value? Example Data: +----+----+----------------------------+---------------------------+ | id|name|precondition |businessTransaction | | 1 |John|{someField1, equals, 100} |{someField1 -\> 100, ...} | | 2 |Jane|{someField2, equals, 200} |{someField2 -\> 150, ...} | +----+----+----------------------------+---------------------------+  In this example, I need to dynamically compare businessTransaction.someField1 with 100 for the first row and businessTransaction.someField2 with 200 for the second row. Any help or guidance on how to achieve this would be greatly appreciated! Here’s a simplified version of my approach: from pyspark.sql.functions import col, concat_ws, expr, when # Create the dynamic field path df_exploded = df_exploded.withColumn( 'dynamic_field_path', concat_ws(".", lit("businessTransaction"), col('precondition.field')) ) # Try to retrieve the value from the dynamically specified field using expr df_exploded = df_exploded.withColumn( 'dynamic_field', expr("dynamic_field_path") ) # Check preconditions df_preconditions_checked = df_exploded.withColumn( "is_matching_precondition", when( col('dynamic_field') == col('precondition.matchingValue'), True ).otherwise(False) ) # Filter distinct _id where is_matching_precondition is True df_matching_preconditions = df_preconditions_checked.filter(col("is_matching_precondition") == True).select(col('_id')).distinct()  Problem: The above code does not work as intended. The dynamic_field column ends up containing the literal string path instead of the actual value from the dynamically specified field. I receive an error indicating that columns are not iterable.

Answer: You can create a udf to handle the dynamic comparison part as follows: from pyspark.sql import SparkSession from pyspark.sql import functions as F from pyspark.sql.types import StructType, StructField, StringType spark = SparkSession.builder.getOrCreate() data = [ ("1", "John", ("someField1", "equals", "100"), {"someField1": "100", "someField2": "120", "someField3": "150"}), ("2", "Jane", ("someField2", "greaterThan", "200"), {"someField1": "100", "someField2": "150", "someField3": "180"}) ] schema = StructType([ StructField("_id", StringType(), True), StructField("name", StringType(), True), StructField("precondition", StructType([ StructField("field", StringType(), True), StructField("matchingType", StringType(), True), StructField("matchingValue", StringType(), True) ]), True), StructField("businessTransaction", StructType([ StructField("someField1", StringType(), True), StructField("someField2", StringType(), True), StructField("someField3", StringType(), True) ]), True) ]) df = spark.createDataFrame(data, schema) df.printSchema() # root # |-- _id: string (nullable = true) # |-- name: string (nullable = true) # |-- precondition: struct (nullable = true) # | |-- field: string (nullable = true) # | |-- matchingType: string (nullable = true) # | |-- matchingValue: string (nullable = true) # |-- businessTransaction: struct (nullable = true) # | |-- someField1: string (nullable = true) # | |-- someField2: string (nullable = true) # | |-- someField3: string (nullable = true) def evaluate_precondition(row): field = row.precondition.field matching_type = row.precondition.matchingType matching_value = row.precondition.matchingValue actual_value = row.businessTransaction[field] if matching_type == "equals": return int(actual_value) == int(matching_value) elif matching_type == "greaterThan": return int(actual_value) > int(matching_value) elif matching_type == "lessThan": return int(actual_value) < int(matching_value) else: return False evaluate_precondition_udf = F.udf(evaluate_precondition) df_preconditions_checked = df.withColumn( "is_matching_precondition", evaluate_precondition_udf(F.struct("precondition", "businessTransaction")) ) df_preconditions_checked.show(truncate=False) # +---+----+------------------------------+-------------------+------------------------+ # |_id|name|precondition |businessTransaction|is_matching_precondition| # +---+----+------------------------------+-------------------+------------------------+ # |1 |John|{someField1, equals, 100} |{100, 120, 150} |true | # |2 |Jane|{someField2, greaterThan, 200}|{100, 150, 180} |false | # +---+----+------------------------------+-------------------+------------------------+ df_matching_preconditions = ( df_preconditions_checked .filter(F.col("is_matching_precondition") == True) .select(F.col("_id")).distinct() ) df_matching_preconditions.show() # +---+ # |_id| # +---+ # | 1| # +---+  To make this work we are passing a new struct column to the UDF and then fetching the values "dynamically" from it, inside the UDF. Also, you can expand the comparison operators as per your choice in the UDF.