Repository: feast-dev/feast

Issue:
Spark 3+ support
## Expected Behavior 

Can use spark 3+ clusters to run jobs.

## Current Behavior

If used together with spark 3+, there is an exception 

```Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/StreamWriteSupport```

which is caused by old (2+) spark dependencies for IngestionJob. 

## Steps to reproduce

I guess `pip install pyspark>=3` and start any job will be enough

### Specifications

- Version: 0.8.0
- Platform: linux
- Subsystem: 

## Possible Solution

Bump spark.version in dependencies and update job code accordingly.



Discussion:
@oavdeev @pyalex I believe there was a specific reason why we couldn't use Spark 3.0 right? Protobuf/grpc dependencies? 

EDIT: So I think the question is whether we can write the job for Spark 2.0 but make it compatible with 3+
FYI https://github.com/RedisLabs/spark-redis/issues/253
Main reason for spark 2.4 was that [bq connector](https://github.com/GoogleCloudDataproc/spark-bigquery-connector) was not supporting 3.x on that moment and that dataproc stable version is still 2.4.
BQ connector seems not an issue anymore

Resolving patch:
diff --git a/sdk/python/feast/pyspark/launchers/aws/emr_utils.py b/sdk/python/feast/pyspark/launchers/aws/emr_utils.py
--- a/sdk/python/feast/pyspark/launchers/aws/emr_utils.py
+++ b/sdk/python/feast/pyspark/launchers/aws/emr_utils.py
@@ -107,7 +107,7 @@ def _sync_offline_to_online_step(
                 "--class",
                 "feast.ingestion.IngestionJob",
                 "--packages",
-                "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.17.2",
+                "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.18.0",
                 jar_path,
             ]
             + args,
@@ -332,7 +332,7 @@ def _stream_ingestion_step(
             + jars_args
             + [
                 "--packages",
-                "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.17.2",
+                "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.18.0",
                 jar_path,
             ]
             + args,
diff --git a/sdk/python/feast/pyspark/launchers/standalone/local.py b/sdk/python/feast/pyspark/launchers/standalone/local.py
--- a/sdk/python/feast/pyspark/launchers/standalone/local.py
+++ b/sdk/python/feast/pyspark/launchers/standalone/local.py
@@ -222,7 +222,7 @@ class StandaloneClusterLauncher(JobLauncher):
     Submits jobs to a standalone Spark cluster in client mode.
     """
 
-    BQ_CONNECTOR_VERSION = "2.12:0.17.3"
+    BQ_CONNECTOR_VERSION = "2.12:0.18.0"
 
     def __init__(self, master_url: str, spark_home: str = None):
         """
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -36,12 +36,12 @@
     "pandas~=1.0.0",
     "pandavro==1.5.*",
     "protobuf>=3.10",
-    "PyYAML==5.1.*",
+    "PyYAML==5.3.*",
     "fastavro>=0.22.11,<0.23",
     "tabulate==0.8.*",
     "toml==0.10.*",
     "tqdm==4.*",
-    "pyarrow<0.16.0,>=0.15.1",
+    "pyarrow==2.0.0",
     "numpy",
     "google",
 ]


Test patch:
diff --git a/spark/ingestion/src/test/scala/feast/ingestion/SparkSpec.scala b/spark/ingestion/src/test/scala/feast/ingestion/SparkSpec.scala
--- a/spark/ingestion/src/test/scala/feast/ingestion/SparkSpec.scala
+++ b/spark/ingestion/src/test/scala/feast/ingestion/SparkSpec.scala
@@ -39,6 +39,7 @@ class SparkSpec extends UnitSpec with BeforeAndAfter {
       )
       .set("spark.metrics.conf.*.sink.statsd.host", "localhost")
       .set("spark.metrics.conf.*.sink.statsd.port", "8125")
+      .set("spark.sql.legacy.allowUntypedScalaUDF", "true")
 
     sparkSession = SparkSession
       .builder()