Bug: Nondeterministic type checker error in data tests
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Dec/21 12:40
Affects Version/s: 3.3.0

Occasionally, I see different errors around {{_jvm}}, {{_wrapped}} and {{_jsparkSession}} popping out in data tests

{code}
E     ../../spark/python/pyspark/sql/catalog:71: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/catalog:72: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/catalog:349: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/catalog:360: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/column:920: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:400: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:407: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:413: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:602: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:604: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:620: error: Cannot determine type of "_jvm"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:621: error: Cannot determine type of "_jvm"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/pandas/conversion:622: error: Cannot determine type of "_wrapped"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/readwriter:115: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/readwriter:307: error: Cannot determine type of "_jvm"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/readwriter:308: error: Cannot determine type of "_jvm"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/session:283: error: Cannot determine type of "_jvm"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/session:284: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/streaming:377: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/udf:225: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/udf:456: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/udf:509: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/udf:510: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
E     ../../spark/python/pyspark/sql/udf:533: error: Cannot determine type of "_jsparkSession"  [has-type] (diff)
{code}

The set of errors is not always the same, so it indicates so issues with type checker.

I am going to add {{casts}} and / or annotations for these lines, so we don't get flaky tests.