Repository: mitchelllisle/sparkdantic

Issue:
mypy complains about spark_type kwarg for Field
Love sparkdantic, and continue to rely on it.  Thank you for your work!

One minor annoyance is that mypy always yells at me for using the `spark_type` kwarg to be explicit about the native spark type that should be used. For example: 

```python
class Product(SparkModel):
    name: Field(description="the name of the product", spark_type=StringType)
    quantity: Field(description="number available", spark_type=LongType)
```

This raises two warnings, one for each property:

```
Unexpected keyword argument "spark_type" for "Field"
Unexpected keyword argument "spark_type" for "Field"
```

This is because, of course, `pydantic.Field` isn't expecting this kwarg.  Currently it seems like sparkdantic relies on pydantic's behavior, which throws any unexpected `kwarg` into `FieldInfo.json_extra_schema`.  https://github.com/mitchelllisle/sparkdantic/blob/083c4f32212da4de76725b5f6d321a6dcfa4d113/src/sparkdantic/model.py#L150

I'm not necessarily arguing against this choice, but I wonder if we could find a way that makes the type checker happy as well.  Perhaps a subclass of `pydantic.Field` like `pydantic.SparkField` that handles sparkdantic-specific kwargs like this?

I'm happy to lend a hand in implementation, too.

Discussion:
@boblannon: for my specific problems with `spark_type`, I find that I can satisfy mypy with a really thin shim:

edit: Turns out `pydantic.Field` is a function with a lot of complex overloading.   had to change my shim to reflect this

```python
from typing import Optional, Type
from pyspark.sql.types import DataType

def SparkField(*args, spark_type: Optional[Type[DataType]] = None, **kwargs) -> Any:
    if spark_type is not None:
        kwargs["spark_type"] = spark_type
    return Field(*args, **kwargs)
```

@mitchelllisle: Hi @boblannon 
Thanks for the suggestion. This makes sense to me and does feel a bit clearer than using `Field` directly. 

I'm happy to implement this suggestion but if you'd like to contribute it directly send me the PR and I'll happily cut a new release.

Thanks again for the suggestion!@mitchelllisle: @boblannon I've put up a PR #546 for this. Happy to wait for you to review / check if fixes your mypy issue before merging@boblannon: Awesome!  I'm sorry I didn't get time to make one myself, but I'll test this out.  I won't get a chance until tomorrow, though.@boblannon: sorry for the delay.  can confirm that this fixed my mypy issue!@mitchelllisle: Have merged it in. Will do a release shortly!@mitchelllisle: Fixed in #546 @boblannon: Thanks so much!

Solution patch:
diff --git a/COVERAGE.txt b/COVERAGE.txt
index 4bb68b06..99823e6c 100644
--- a/COVERAGE.txt
+++ b/COVERAGE.txt
@@ -1,6 +1,6 @@
 Name                          Stmts   Miss  Cover
 -------------------------------------------------
 src/sparkdantic/__init__.py       4      0   100%
-src/sparkdantic/model.py        128      1    99%
+src/sparkdantic/model.py        131      1    99%
 -------------------------------------------------
-TOTAL                           132      1    99%
+TOTAL                           135      1    99%
diff --git a/src/sparkdantic/__init__.py b/src/sparkdantic/__init__.py
index b05f044c..0b4682d5 100644
--- a/src/sparkdantic/__init__.py
+++ b/src/sparkdantic/__init__.py
@@ -2,4 +2,4 @@
 __author__ = 'Mitchell Lisle'
 __email__ = 'm.lisle90@gmail.com'
 
-from sparkdantic.model import SparkModel, create_spark_schema
+from sparkdantic.model import SparkField, SparkModel, create_spark_schema
diff --git a/src/sparkdantic/model.py b/src/sparkdantic/model.py
index 75ff68f4..83d5da8d 100644
--- a/src/sparkdantic/model.py
+++ b/src/sparkdantic/model.py
@@ -6,11 +6,11 @@
 from decimal import Decimal
 from enum import Enum
 from types import MappingProxyType
-from typing import Annotated, Tuple, Type, Union, get_args, get_origin
+from typing import Annotated, Optional, Tuple, Type, Union, get_args, get_origin
 from uuid import UUID
 
 from annotated_types import BaseMetadata
-from pydantic import AliasChoices, AliasPath, BaseModel, ConfigDict, SecretBytes, SecretStr
+from pydantic import AliasChoices, AliasPath, BaseModel, ConfigDict, Field, SecretBytes, SecretStr
 from pydantic.fields import FieldInfo
 from pydantic.json_schema import JsonSchemaMode
 from pyspark.sql import functions as F  # noqa
@@ -85,6 +85,12 @@
 BaseModelOrSparkModel = Union[BaseModel, 'SparkModel']
 
 
+def SparkField(*args, spark_type: Optional[Type[DataType]] = None, **kwargs) -> Field:
+    if spark_type is not None:
+        kwargs['spark_type'] = spark_type
+    return Field(*args, **kwargs)
+
+
 class SparkModel(BaseModel):
     """Spark Model representing a Pydantic BaseModel with additional methods to convert it to a PySpark schema.
 


Test patch:
diff --git a/tests/test_spark_field.py b/tests/test_spark_field.py
new file mode 100644
index 00000000..fafd53e1
--- /dev/null
+++ b/tests/test_spark_field.py
@@ -0,0 +1,18 @@
+from pyspark.sql import types as T
+
+from sparkdantic import SparkField, SparkModel
+
+
+class Model(SparkModel):
+    z: int = SparkField(spark_type=T.StringType, alias='_z')
+    zz: int
+
+
+def test_spark_field():
+    schema = Model.model_spark_schema()
+    assert schema == T.StructType(
+        [
+            T.StructField('_z', T.StringType(), False),
+            T.StructField('zz', T.IntegerType(), False),
+        ]
+    )