Repository: mitchelllisle/sparkdantic

Issue:
[Suggestion]: Make `pyspark` an optional dependency
Hi 👋

We are currently experimenting with using `sparkdantic` on our Spark schema definitions in our pipelines inside Databricks. However, based on our current configuration, we are bound to installing all dependencies inside notebook scopes - rather than installing dependencies on the cluster-level. This means we need to run the `!pip install` command for each dependency used in the beginning of our notebooks.

We've noticed that the `sparkdantic` installation is taking a lot of time to install as it also installs `pyspark` as part of its dependencies, even though `spark` is already available inside the Databricks environment. A potential solution for this is to move `pyspark` to become an optional dependency, rather than a mandatory dependency.

Any thoughts on this suggestion?

Discussion:
@jaceklaskowski: I've noticed it too lately and would much appreciate this change ❤️ @mitchstockdale: Hey @FredrikBakken, thanks for the suggestion, and apologies for the delayed response.

I’m in favour of this and am happy to implement it in the near future if no one else gets around to it.@mitchelllisle: Sorry for the delay getting back on this one. Haven't had much time to work on this lately but I'm ok with this approach since spark is often a provided dependency in many other implementations.

@mitchstockdale I'll have some time this week to do this and want to bundle in another change raised but if you get around to it first let me know and I'll cut a new release.

Thanks for the suggestions and appreciate the patience!@mitchstockdale: @mitchelllisle - I haven't had a look at this yet but I was thinking about it over the last week or so, and have an additional proposal to enhance it #536 @mitchelllisle: This has been released in most recent release [v2.0.0](https://github.com/mitchelllisle/sparkdantic/releases/tag/v2.0.0)

Solution patch:
diff --git a/COVERAGE.txt b/COVERAGE.txt
index 0cac473..a50d451 100644
--- a/COVERAGE.txt
+++ b/COVERAGE.txt
@@ -1,7 +1,8 @@
 Name                            Stmts   Miss  Cover
 ---------------------------------------------------
 src/sparkdantic/__init__.py         4      0   100%
-src/sparkdantic/exceptions.py       1      0   100%
-src/sparkdantic/model.py          122      0   100%
+src/sparkdantic/exceptions.py       2      0   100%
+src/sparkdantic/model.py          144      3    98%
+src/sparkdantic/utils.py           19      3    84%
 ---------------------------------------------------
-TOTAL                             127      0   100%
+TOTAL                             169      6    96%
diff --git a/Makefile b/Makefile
index b2acc0a..6ea987b 100644
--- a/Makefile
+++ b/Makefile
@@ -73,13 +73,13 @@ install-tests:
 	poetry install --only test
 
 install-all:
-	poetry install --with dev,test,docs
+	poetry install --with dev,test,docs -E pyspark
 
 install-dev-local: ## install all the stuff you need to develop locally
 	pip install --upgrade pip
 	pip install wheel
 	pip install -e .
-	poetry install --with dev,test,docs
+	poetry install --with dev,test,docs -E pyspark
 	pre-commit install
 
 publish: dist ## publish the package to PyPI
diff --git a/README.md b/README.md
index 24d34fd..1eeb0fe 100644
--- a/README.md
+++ b/README.md
@@ -14,8 +14,29 @@ named `SparkModel` that extends the Pydantic's `BaseModel`.
 
 ## Features
 
-- Conversion from Pydantic model to PySpark schema
+- Conversion from Pydantic model to PySpark schema (`StructType` or JSON)
 - Type coercion
+- PySpark as an optional dependency
+
+## Installation
+
+Without PySpark:
+
+```shell
+pip install sparkdantic
+```
+
+> Note: only **JSON** schema generation features are available without PySpark installed. If you attempt to use PySpark-dependent schema generation features, SparkDantic will check that a supported version of Pyspark is installed.
+
+With PySpark:
+
+```shell
+pip install "sparkdantic[pyspark]"
+```
+
+### Supported PySpark versions
+
+`3.3.0` or higher, but not `4.0`
 
 ## Usage
 
@@ -50,7 +71,7 @@ class MyEnumModel(SparkModel):
 
 #### Using `SparkModel`
 
-Pydantic has existing models for generating json schemas (with `model_json_schema`). With a `SparkModel` you can 
+Pydantic has existing models for generating [JSON schemas](https://docs.pydantic.dev/2.10/concepts/json_schema/) (with `model_json_schema`). With a `SparkModel` you can 
 generate a PySpark schema from the model fields using the `model_spark_schema()` method:
 
 ```python
@@ -67,12 +88,50 @@ StructType([
 ])
 ```
 
+You can also generate a [PySpark-compatible JSON schema](https://spark.apache.org/docs/3.5.4/api/python/reference/pyspark.sql/api/pyspark.sql.types.StructType.html#pyspark.sql.types.StructType.fromJson) from the model fields using the `model_json_spark_schema` method:
+
+```python
+json_spark_schema = MyModel.model_json_spark_schema()
+```
+
+Provides this schema:
+
+```python
+{
+    "type": "struct",
+    "fields": [
+        {
+            "name": "name",
+            "type": "string",
+            "nullable": False,
+            "metadata": {}
+        },
+        {
+            "name": "age",
+            "type": "integer",
+            "nullable": False,
+            "metadata": {}
+        },
+        {
+            "name": "hobbies",
+            "type": {
+                "type": "array",
+                "elementType": "string",
+                "containsNull": False
+            },
+            "nullable": False,
+            "metadata": {}
+        }
+    ]
+}
+```
+
 #### Using Pydantic `BaseModel`
 
 You can also generate a PySpark schema for existing Pydantic models using the `create_spark_schema` function:
 
 ```python
-from sparkdantic import create_spark_schema
+from sparkdantic import create_spark_schema, create_json_spark_schema
 
 class EmployeeModel(BaseModel):
     id: int
@@ -81,11 +140,12 @@ class EmployeeModel(BaseModel):
     department_code: str
 
 spark_schema = create_spark_schema(EmployeeModel)
+json_spark_schema = create_json_spark_schema(EmployeeModel)
 ```
 
 > ℹ️  In addition to the automatic type conversion, you can also explicitly coerce data types to Spark native types by 
->  setting the `spark_type` attribute in the `SparkField` function (which extends the Pydantic `Field` function), like so: `SparkField(spark_type=DataType)`.
->  Please replace DataType with the actual Spark data type you want to use.
+>  setting the `spark_type` attribute in the `SparkField` function (which extends the Pydantic `Field` function), like so: `SparkField(spark_type=<datatype>)`.
+>  `datatype` accepts `str` (e.g. `"bigint"`) or `pyspark.sql.types.DataType` (e.g. `LongType`).
 >  This is useful when you want to use a specific data type then the one that Sparkdantic infers by default. 
 
 ## Contributing
diff --git a/poetry.lock b/poetry.lock
index c2025f0..1861f07 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.5 and should not be changed by hand.
+# This file is automatically @generated by Poetry 2.0.0 and should not be changed by hand.
 
 [[package]]
 name = "annotated-types"
@@ -6,6 +6,7 @@ version = "0.7.0"
 description = "Reusable constraint types to use with typing.Annotated"
 optional = false
 python-versions = ">=3.8"
+groups = ["main"]
 files = [
     {file = "annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53"},
     {file = "annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89"},
@@ -17,6 +18,7 @@ version = "23.1.0"
 description = "Classes Without Boilerplate"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev", "test"]
 files = [
     {file = "attrs-23.1.0-py3-none-any.whl", hash = "sha256:1f28b4522cdc2fb4256ac1a020c78acf9cba2c6b461ccd2c126f3aa8e8335d04"},
     {file = "attrs-23.1.0.tar.gz", hash = "sha256:6279836d581513a26f1bf235f9acd333bc9115683f14f7e8fae46c98fc50e015"},
@@ -35,6 +37,7 @@ version = "2.12.1"
 description = "Internationalization utilities"
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "Babel-2.12.1-py3-none-any.whl", hash = "sha256:b4246fb7677d3b98f501a39d43396d3cafdc8eadb045f4a31be01863f655c610"},
     {file = "Babel-2.12.1.tar.gz", hash = "sha256:cc2d99999cd01d44420ae725a21c9e3711b3aadc7976d6147f622d8581963455"},
@@ -46,6 +49,7 @@ version = "6.0.0"
 description = "An easy safelist-based HTML-sanitizing tool."
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
 files = [
     {file = "bleach-6.0.0-py3-none-any.whl", hash = "sha256:33c16e3353dbd13028ab4799a0f89a83f113405c766e9c122df8a06f5b85b3f4"},
     {file = "bleach-6.0.0.tar.gz", hash = "sha256:1a1a85c1595e07d8db14c5f09f09e6433502c51c595970edc090551f0db99414"},
@@ -64,6 +68,7 @@ version = "1.0.1"
 description = "Version-bump your software with a single command!"
 optional = false
 python-versions = ">=3.5"
+groups = ["dev"]
 files = [
     {file = "bump2version-1.0.1-py2.py3-none-any.whl", hash = "sha256:37f927ea17cde7ae2d7baf832f8e80ce3777624554a653006c9144f8017fe410"},
     {file = "bump2version-1.0.1.tar.gz", hash = "sha256:762cb2bfad61f4ec8e2bdf452c7c267416f8c70dd9ecb1653fd0bbb01fa936e6"},
@@ -75,6 +80,7 @@ version = "0.6.0"
 description = "Version-bump your software with a single command!"
 optional = false
 python-versions = "*"
+groups = ["dev"]
 files = [
     {file = "bumpversion-0.6.0-py2.py3-none-any.whl", hash = "sha256:4eb3267a38194d09f048a2179980bb4803701969bff2c85fa8f6d1ce050be15e"},
     {file = "bumpversion-0.6.0.tar.gz", hash = "sha256:4ba55e4080d373f80177b4dabef146c07ce73c7d1377aabf9d3c3ae1f94584a6"},
@@ -89,6 +95,7 @@ version = "2024.7.4"
 description = "Python package for providing Mozilla's CA Bundle."
 optional = false
 python-versions = ">=3.6"
+groups = ["dev", "docs"]
 files = [
     {file = "certifi-2024.7.4-py3-none-any.whl", hash = "sha256:c198e21b1289c2ab85ee4e67bb4b4ef3ead0892059901a8d5b622f24a1101e90"},
     {file = "certifi-2024.7.4.tar.gz", hash = "sha256:5a1e7645bc0ec61a09e26c36f6106dd4cf40c6db3a1fb6352b0244e7fb057c7b"},
@@ -100,6 +107,8 @@ version = "1.15.1"
 description = "Foreign Function Interface for Python calling C code."
 optional = false
 python-versions = "*"
+groups = ["dev"]
+markers = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and sys_platform == \"linux\" and platform_python_implementation != \"PyPy\""
 files = [
     {file = "cffi-1.15.1-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:a66d3508133af6e8548451b25058d5812812ec3798c886bf38ed24a98216fab2"},
     {file = "cffi-1.15.1-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:470c103ae716238bbe698d67ad020e1db9d9dba34fa5a899b5e21577e6d52ed2"},
@@ -176,6 +185,7 @@ version = "3.3.1"
 description = "Validate configuration and produce human readable error messages."
 optional = false
 python-versions = ">=3.6.1"
+groups = ["dev"]
 files = [
     {file = "cfgv-3.3.1-py2.py3-none-any.whl", hash = "sha256:c6a0883f3917a037485059700b9e75da2464e6c27051014ad85ba6aaa5884426"},
     {file = "cfgv-3.3.1.tar.gz", hash = "sha256:f5a830efb9ce7a445376bb66ec94c638a9787422f96264c98edc6bdeed8ab736"},
@@ -187,6 +197,7 @@ version = "3.2.0"
 description = "The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet."
 optional = false
 python-versions = ">=3.7.0"
+groups = ["dev", "docs"]
 files = [
     {file = "charset-normalizer-3.2.0.tar.gz", hash = "sha256:3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace"},
     {file = "charset_normalizer-3.2.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:0b87549028f680ca955556e3bd57013ab47474c3124dc069faa0b6545b6c9710"},
@@ -271,6 +282,7 @@ version = "8.1.6"
 description = "Composable command line interface toolkit"
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "click-8.1.6-py3-none-any.whl", hash = "sha256:fa244bb30b3b5ee2cae3da8f55c9e5e0c0e86093306301fb418eb9dc40fbded5"},
     {file = "click-8.1.6.tar.gz", hash = "sha256:48ee849951919527a045bfe3bf7baa8a959c423134e1a5b98c05c20ba75a1cbd"},
@@ -285,10 +297,12 @@ version = "0.4.6"
 description = "Cross-platform colored terminal text."
 optional = false
 python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7"
+groups = ["docs", "test"]
 files = [
     {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"},
     {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"},
 ]
+markers = {test = "sys_platform == \"win32\""}
 
 [[package]]
 name = "coverage"
@@ -296,6 +310,7 @@ version = "7.6.4"
 description = "Code coverage measurement for Python"
 optional = false
 python-versions = ">=3.9"
+groups = ["test"]
 files = [
     {file = "coverage-7.6.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5f8ae553cba74085db385d489c7a792ad66f7f9ba2ee85bfa508aeb84cf0ba07"},
     {file = "coverage-7.6.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8165b796df0bd42e10527a3f493c592ba494f16ef3c8b531288e3d0d72c1f6f0"},
@@ -373,6 +388,8 @@ version = "43.0.1"
 description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
+markers = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and sys_platform == \"linux\""
 files = [
     {file = "cryptography-43.0.1-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:8385d98f6a3bf8bb2d65a73e17ed87a3ba84f6991c155691c51112075f9ffc5d"},
     {file = "cryptography-43.0.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:27e613d7077ac613e399270253259d9d53872aaf657471473ebfc9a52935c062"},
@@ -422,6 +439,7 @@ version = "0.3.7"
 description = "Distribution utilities"
 optional = false
 python-versions = "*"
+groups = ["dev"]
 files = [
     {file = "distlib-0.3.7-py2.py3-none-any.whl", hash = "sha256:2e24928bc811348f0feb63014e97aaae3037f2cf48712d51ae61df7fd6075057"},
     {file = "distlib-0.3.7.tar.gz", hash = "sha256:9dafe54b34a028eafd95039d5e5d4851a13734540f1331060d31c9916e7147a8"},
@@ -433,6 +451,7 @@ version = "0.16.0"
 description = "Dlint is a tool for encouraging best coding practices and helping ensure Python code is secure."
 optional = false
 python-versions = "<4.0.0,>=3.8.1"
+groups = ["dev"]
 files = [
     {file = "dlint-0.16.0-py3-none-any.whl", hash = "sha256:b4b3b01460d1b310c3f2356be9ce9b7bc2112dfd9be1842eefef36a54eac9443"},
 ]
@@ -446,6 +465,7 @@ version = "0.20.1"
 description = "Docutils -- Python Documentation Utilities"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
 files = [
     {file = "docutils-0.20.1-py3-none-any.whl", hash = "sha256:96f387a2c5562db4476f09f13bbab2192e764cac08ebbf3a34a95d9b1e4a59d6"},
     {file = "docutils-0.20.1.tar.gz", hash = "sha256:f08a4e276c3a1583a86dce3e34aba3fe04d02bba2dd51ed16106244e8a923e3b"},
@@ -457,6 +477,8 @@ version = "1.1.2"
 description = "Backport of PEP 654 (exception groups)"
 optional = false
 python-versions = ">=3.7"
+groups = ["test"]
+markers = "python_version < \"3.11\""
 files = [
     {file = "exceptiongroup-1.1.2-py3-none-any.whl", hash = "sha256:e346e69d186172ca7cf029c8c1d16235aa0e04035e5750b4b95039e65204328f"},
     {file = "exceptiongroup-1.1.2.tar.gz", hash = "sha256:12c3e887d6485d16943a309616de20ae5582633e0a2eda17f4e10fd61c1e8af5"},
@@ -471,6 +493,7 @@ version = "36.1.1"
 description = "Faker is a Python package that generates fake data for you."
 optional = false
 python-versions = ">=3.9"
+groups = ["test"]
 files = [
     {file = "Faker-36.1.1-py3-none-any.whl", hash = "sha256:ad1f1be7fd692ec0256517404a9d7f007ab36ac5d4674082fa72404049725eaa"},
     {file = "faker-36.1.1.tar.gz", hash = "sha256:7cb2bbd4c8f040e4a340ae4019e9a48b6cf1db6a71bda4e5a61d8d13b7bef28d"},
@@ -485,6 +508,7 @@ version = "3.12.2"
 description = "A platform independent file lock."
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
 files = [
     {file = "filelock-3.12.2-py3-none-any.whl", hash = "sha256:cbb791cdea2a72f23da6ac5b5269ab0a0d161e9ef0100e653b69049a7706d1ec"},
     {file = "filelock-3.12.2.tar.gz", hash = "sha256:002740518d8aa59a26b0c76e10fb8c6e15eae825d34b6fdf670333fd7b938d81"},
@@ -500,6 +524,7 @@ version = "7.1.1"
 description = "the modular source code checker: pep8 pyflakes and co"
 optional = false
 python-versions = ">=3.8.1"
+groups = ["dev"]
 files = [
     {file = "flake8-7.1.1-py2.py3-none-any.whl", hash = "sha256:597477df7860daa5aa0fdd84bf5208a043ab96b8e96ab708770ae0364dd03213"},
     {file = "flake8-7.1.1.tar.gz", hash = "sha256:049d058491e228e03e67b390f311bbf88fce2dbaa8fa673e7aea87b7198b8d38"},
@@ -516,6 +541,7 @@ version = "24.12.12"
 description = "A plugin for flake8 finding likely bugs and design problems in your program. Contains warnings that don't belong in pyflakes and pycodestyle."
 optional = false
 python-versions = ">=3.8.1"
+groups = ["dev"]
 files = [
     {file = "flake8_bugbear-24.12.12-py3-none-any.whl", hash = "sha256:1b6967436f65ca22a42e5373aaa6f2d87966ade9aa38d4baf2a1be550767545e"},
     {file = "flake8_bugbear-24.12.12.tar.gz", hash = "sha256:46273cef0a6b6ff48ca2d69e472f41420a42a46e24b2a8972e4f0d6733d12a64"},
@@ -534,6 +560,7 @@ version = "2.1.0"
 description = "Copy your docs directly to the gh-pages branch."
 optional = false
 python-versions = "*"
+groups = ["docs"]
 files = [
     {file = "ghp-import-2.1.0.tar.gz", hash = "sha256:9c535c4c61193c2df8871222567d7fd7e5014d835f97dc7b7439069e2413d343"},
     {file = "ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619"},
@@ -551,6 +578,7 @@ version = "0.32.3"
 description = "Signatures for entire Python programs. Extract the structure, the frame, the skeleton of your project, to generate API documentation or find breaking changes in your API."
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "griffe-0.32.3-py3-none-any.whl", hash = "sha256:d9471934225818bf8f309822f70451cc6abb4b24e59e0bb27402a45f9412510f"},
     {file = "griffe-0.32.3.tar.gz", hash = "sha256:14983896ad581f59d5ad7b6c9261ff12bdaa905acccc1129341d13e545da8521"},
@@ -565,6 +593,7 @@ version = "6.125.3"
 description = "A library for property-based testing"
 optional = false
 python-versions = ">=3.9"
+groups = ["test"]
 files = [
     {file = "hypothesis-6.125.3-py3-none-any.whl", hash = "sha256:ae0381987d0ccacc62867343acc38a0ca5e959fc4da98b3e018b0debf080a557"},
     {file = "hypothesis-6.125.3.tar.gz", hash = "sha256:6276d9d8f7f91ebd7ad95081f004bcf5699afb4d680a8321861cbf3a5de45d3b"},
@@ -598,6 +627,7 @@ version = "1.5.0"
 description = "A tool for generating OIDC identities"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "id-1.5.0-py3-none-any.whl", hash = "sha256:f1434e1cef91f2cbb8a4ec64663d5a23b9ed43ef44c4c957d02583d61714c658"},
     {file = "id-1.5.0.tar.gz", hash = "sha256:292cb8a49eacbbdbce97244f47a97b4c62540169c976552e497fd57df0734c1d"},
@@ -617,6 +647,7 @@ version = "2.5.25"
 description = "File identification library for Python"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "identify-2.5.25-py2.py3-none-any.whl", hash = "sha256:9df2489842707d431b38ce3410ef8df40da5b10a3e28a3fcac1a42523e956409"},
     {file = "identify-2.5.25.tar.gz", hash = "sha256:db4de0e758c0db8f81996816cd2f3f2f8c5c8d49a7fd02f3b4109aac6fd80e29"},
@@ -631,6 +662,7 @@ version = "3.7"
 description = "Internationalized Domain Names in Applications (IDNA)"
 optional = false
 python-versions = ">=3.5"
+groups = ["dev", "docs"]
 files = [
     {file = "idna-3.7-py3-none-any.whl", hash = "sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0"},
     {file = "idna-3.7.tar.gz", hash = "sha256:028ff3aadf0609c1fd278d8ea3089299412a7a8b9bd005dd08b9f8285bcb5cfc"},
@@ -642,10 +674,12 @@ version = "6.8.0"
 description = "Read metadata from Python packages"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev", "docs"]
 files = [
     {file = "importlib_metadata-6.8.0-py3-none-any.whl", hash = "sha256:3ebb78df84a805d7698245025b975d9d67053cd94c79245ba4b3eb694abe68bb"},
     {file = "importlib_metadata-6.8.0.tar.gz", hash = "sha256:dbace7892d8c0c4ac1ad096662232f831d4e64f4c4545bd53016a3e9d4654743"},
 ]
+markers = {dev = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and python_version < \"3.12\" or python_version < \"3.10\"", docs = "python_version < \"3.10\""}
 
 [package.dependencies]
 zipp = ">=0.5"
@@ -661,6 +695,7 @@ version = "2.0.0"
 description = "brain-dead simple config-ini parsing"
 optional = false
 python-versions = ">=3.7"
+groups = ["test"]
 files = [
     {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"},
     {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"},
@@ -672,6 +707,8 @@ version = "3.3.0"
 description = "Utility functions for Python class constructs"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
+markers = "platform_machine != \"ppc64le\" and platform_machine != \"s390x\""
 files = [
     {file = "jaraco.classes-3.3.0-py3-none-any.whl", hash = "sha256:10afa92b6743f25c0cf5f37c6bb6e18e2c5bb84a16527ccfc0040ea377e7aaeb"},
     {file = "jaraco.classes-3.3.0.tar.gz", hash = "sha256:c063dd08e89217cee02c8d5e5ec560f2c8ce6cdc2fcdc2e68f7b2e5547ed3621"},
@@ -690,6 +727,8 @@ version = "0.8.0"
 description = "Low-level, pure Python DBus protocol wrapper."
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
+markers = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and sys_platform == \"linux\""
 files = [
     {file = "jeepney-0.8.0-py3-none-any.whl", hash = "sha256:c0a454ad016ca575060802ee4d590dd912e35c122fa04e70306de3d076cce755"},
     {file = "jeepney-0.8.0.tar.gz", hash = "sha256:5efe48d255973902f6badc3ce55e2aa6c5c3b3bc642059ef3a91247bcfcc5806"},
@@ -705,6 +744,7 @@ version = "3.1.5"
 description = "A very fast and expressive template engine."
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "jinja2-3.1.5-py3-none-any.whl", hash = "sha256:aba0f4dc9ed8013c424088f68a5c226f7d6097ed89b246d7749c2ec4175c6adb"},
     {file = "jinja2-3.1.5.tar.gz", hash = "sha256:8fefff8dc3034e27bb80d67c671eb8a9bc424c0ef4c0826edbff304cceff43bb"},
@@ -722,6 +762,7 @@ version = "1.0.1"
 description = "JSON Matching Expressions"
 optional = false
 python-versions = ">=3.7"
+groups = ["main"]
 files = [
     {file = "jmespath-1.0.1-py3-none-any.whl", hash = "sha256:02e2e4cc71b5bcab88332eebf907519190dd9e6e82107fa7f83b1003a6252980"},
     {file = "jmespath-1.0.1.tar.gz", hash = "sha256:90261b206d6defd58fdd5e85f478bf633a2901798906be2ad389150c5c60edbe"},
@@ -733,6 +774,8 @@ version = "24.2.0"
 description = "Store and access your passwords safely."
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
+markers = "platform_machine != \"ppc64le\" and platform_machine != \"s390x\""
 files = [
     {file = "keyring-24.2.0-py3-none-any.whl", hash = "sha256:4901caaf597bfd3bbd78c9a0c7c4c29fcd8310dab2cffefe749e916b6527acd6"},
     {file = "keyring-24.2.0.tar.gz", hash = "sha256:ca0746a19ec421219f4d713f848fa297a661a8a8c1504867e55bfb5e09091509"},
@@ -756,6 +799,7 @@ version = "3.6"
 description = "Python implementation of John Gruber's Markdown."
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "Markdown-3.6-py3-none-any.whl", hash = "sha256:48f276f4d8cfb8ce6527c8f79e2ee29708508bf4d40aa410fbc3b4ee832c850f"},
     {file = "Markdown-3.6.tar.gz", hash = "sha256:ed4f41f6daecbeeb96e576ce414c41d2d876daa9a16cb35fa8ed8c2ddfad0224"},
@@ -774,6 +818,7 @@ version = "3.0.0"
 description = "Python port of markdown-it. Markdown parsing, done right!"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "markdown-it-py-3.0.0.tar.gz", hash = "sha256:e3f60a94fa066dc52ec76661e37c851cb232d92f9886b15cb560aaada2df8feb"},
     {file = "markdown_it_py-3.0.0-py3-none-any.whl", hash = "sha256:355216845c60bd96232cd8d8c40e8f9765cc86f46880e43a8fd22dc1a1a8cab1"},
@@ -798,6 +843,7 @@ version = "2.1.3"
 description = "Safely add untrusted strings to HTML/XML markup."
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "MarkupSafe-2.1.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cd0f502fe016460680cd20aaa5a76d241d6f35a1c3350c474bac1273803893fa"},
     {file = "MarkupSafe-2.1.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e09031c87a1e51556fdcb46e5bd4f59dfb743061cf93c4d6831bf894f125eb57"},
@@ -867,6 +913,7 @@ version = "0.7.0"
 description = "McCabe checker, plugin for flake8"
 optional = false
 python-versions = ">=3.6"
+groups = ["dev"]
 files = [
     {file = "mccabe-0.7.0-py2.py3-none-any.whl", hash = "sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e"},
     {file = "mccabe-0.7.0.tar.gz", hash = "sha256:348e0240c33b60bbdf4e523192ef919f28cb2c3d7d5c7794f74009290f236325"},
@@ -878,6 +925,7 @@ version = "0.1.2"
 description = "Markdown URL utilities"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
 files = [
     {file = "mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8"},
     {file = "mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba"},
@@ -889,6 +937,7 @@ version = "1.3.4"
 description = "A deep merge function for 🐍."
 optional = false
 python-versions = ">=3.6"
+groups = ["docs"]
 files = [
     {file = "mergedeep-1.3.4-py3-none-any.whl", hash = "sha256:70775750742b25c0d8f36c55aed03d24c3384d17c951b3175d898bd778ef0307"},
     {file = "mergedeep-1.3.4.tar.gz", hash = "sha256:0096d52e9dad9939c3d975a774666af186eda617e6ca84df4c94dec30004f2a8"},
@@ -900,6 +949,7 @@ version = "0.2.0"
 description = "AutoDoc for MarkDown"
 optional = false
 python-versions = ">=3.6"
+groups = ["docs"]
 files = [
     {file = "mkautodoc-0.2.0.tar.gz", hash = "sha256:b6e0c89804ba39d453ce5795eb040e5615756f90438ac04f47e97893a86a10cd"},
 ]
@@ -913,6 +963,7 @@ version = "1.6.1"
 description = "Project documentation with Markdown."
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "mkdocs-1.6.1-py3-none-any.whl", hash = "sha256:db91759624d1647f3f34aa0c3f327dd2601beae39a366d6e064c03468d35c20e"},
     {file = "mkdocs-1.6.1.tar.gz", hash = "sha256:7b432f01d928c084353ab39c57282f29f92136665bdd6abf7c1ec8d822ef86f2"},
@@ -944,6 +995,7 @@ version = "1.3.0"
 description = "Automatically link across pages in MkDocs."
 optional = false
 python-versions = ">=3.9"
+groups = ["docs"]
 files = [
     {file = "mkdocs_autorefs-1.3.0-py3-none-any.whl", hash = "sha256:d180f9778a04e78b7134e31418f238bba56f56d6a8af97873946ff661befffb3"},
     {file = "mkdocs_autorefs-1.3.0.tar.gz", hash = "sha256:6867764c099ace9025d6ac24fd07b85a98335fbd30107ef01053697c8f46db61"},
@@ -960,6 +1012,7 @@ version = "0.5.0"
 description = "MkDocs plugin to programmatically generate documentation pages during the build"
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "mkdocs_gen_files-0.5.0-py3-none-any.whl", hash = "sha256:7ac060096f3f40bd19039e7277dd3050be9a453c8ac578645844d4d91d7978ea"},
     {file = "mkdocs_gen_files-0.5.0.tar.gz", hash = "sha256:4c7cf256b5d67062a788f6b1d035e157fc1a9498c2399be9af5257d4ff4d19bc"},
@@ -974,6 +1027,7 @@ version = "0.2.0"
 description = "MkDocs extension that lists all dependencies according to a mkdocs.yml file"
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "mkdocs_get_deps-0.2.0-py3-none-any.whl", hash = "sha256:2bf11d0b133e77a0dd036abeeb06dec8775e46efa526dc70667d8863eefc6134"},
     {file = "mkdocs_get_deps-0.2.0.tar.gz", hash = "sha256:162b3d129c7fad9b19abfdcb9c1458a651628e4b1dea628ac68790fb3061c60c"},
@@ -991,6 +1045,7 @@ version = "0.6.1"
 description = "MkDocs plugin to specify the navigation in Markdown instead of YAML"
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "mkdocs_literate_nav-0.6.1-py3-none-any.whl", hash = "sha256:e70bdc4a07050d32da79c0b697bd88e9a104cf3294282e9cb20eec94c6b0f401"},
     {file = "mkdocs_literate_nav-0.6.1.tar.gz", hash = "sha256:78a7ab6d878371728acb0cdc6235c9b0ffc6e83c997b037f4a5c6ff7cef7d759"},
@@ -1005,6 +1060,7 @@ version = "9.6.4"
 description = "Documentation that simply works"
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "mkdocs_material-9.6.4-py3-none-any.whl", hash = "sha256:414e8376551def6d644b8e6f77226022868532a792eb2c9accf52199009f568f"},
     {file = "mkdocs_material-9.6.4.tar.gz", hash = "sha256:4d1d35e1c1d3e15294cb7fa5d02e0abaee70d408f75027dc7be6e30fb32e6867"},
@@ -1034,6 +1090,7 @@ version = "1.3"
 description = "Extension pack for Python Markdown and MkDocs Material."
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "mkdocs_material_extensions-1.3-py3-none-any.whl", hash = "sha256:0297cc48ba68a9fdd1ef3780a3b41b534b0d0df1d1181a44676fda5f464eeadc"},
     {file = "mkdocs_material_extensions-1.3.tar.gz", hash = "sha256:f0446091503acb110a7cab9349cbc90eeac51b58d1caa92a704a81ca1e24ddbd"},
@@ -1045,6 +1102,7 @@ version = "0.28.1"
 description = "Automatic documentation from sources, for MkDocs."
 optional = false
 python-versions = ">=3.9"
+groups = ["docs"]
 files = [
     {file = "mkdocstrings-0.28.1-py3-none-any.whl", hash = "sha256:a5878ae5cd1e26f491ff084c1f9ab995687d52d39a5c558e9b7023d0e4e0b740"},
     {file = "mkdocstrings-0.28.1.tar.gz", hash = "sha256:fb64576906771b7701e8e962fd90073650ff689e95eb86e86751a66d65ab4489"},
@@ -1073,6 +1131,7 @@ version = "1.2.1"
 description = "A Python handler for mkdocstrings."
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "mkdocstrings_python-1.2.1-py3-none-any.whl", hash = "sha256:7c08f33e9ba7b1655e9cf0608eba3ce7a9513bd8b42a68a8d24ffaf4a6a50cfc"},
     {file = "mkdocstrings_python-1.2.1.tar.gz", hash = "sha256:ae40825b3b676a94626882901ed9c8fcf9a7f0330e466ffe37ce15c525987aa9"},
@@ -1088,6 +1147,8 @@ version = "9.1.0"
 description = "More routines for operating on iterables, beyond itertools"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
+markers = "platform_machine != \"ppc64le\" and platform_machine != \"s390x\""
 files = [
     {file = "more-itertools-9.1.0.tar.gz", hash = "sha256:cabaa341ad0389ea83c17a94566a53ae4c9d07349861ecb14dc6d0345cf9ac5d"},
     {file = "more_itertools-9.1.0-py3-none-any.whl", hash = "sha256:d2bc7f02446e86a68911e58ded76d6561eea00cddfb2a91e7019bbb586c799f3"},
@@ -1099,6 +1160,7 @@ version = "1.15.0"
 description = "Optional static typing for Python"
 optional = false
 python-versions = ">=3.9"
+groups = ["dev"]
 files = [
     {file = "mypy-1.15.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:979e4e1a006511dacf628e36fadfecbcc0160a8af6ca7dad2f5025529e082c13"},
     {file = "mypy-1.15.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c4bb0e1bd29f7d34efcccd71cf733580191e9a264a2202b0239da95984c5b559"},
@@ -1152,6 +1214,7 @@ version = "1.0.0"
 description = "Type system extensions for programs checked with the mypy type checker."
 optional = false
 python-versions = ">=3.5"
+groups = ["dev"]
 files = [
     {file = "mypy_extensions-1.0.0-py3-none-any.whl", hash = "sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d"},
     {file = "mypy_extensions-1.0.0.tar.gz", hash = "sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782"},
@@ -1163,6 +1226,7 @@ version = "1.8.0"
 description = "Node.js virtual environment builder"
 optional = false
 python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"
+groups = ["dev"]
 files = [
     {file = "nodeenv-1.8.0-py2.py3-none-any.whl", hash = "sha256:df865724bb3c3adc86b3876fa209771517b0cfe596beff01a92700e0e8be4cec"},
     {file = "nodeenv-1.8.0.tar.gz", hash = "sha256:d51e0c37e64fbf47d017feac3145cdbb58836d7eee8c6f6d3b6880c5456227d2"},
@@ -1177,6 +1241,7 @@ version = "24.2"
 description = "Core utilities for Python packages"
 optional = false
 python-versions = ">=3.8"
+groups = ["main", "dev", "docs", "test"]
 files = [
     {file = "packaging-24.2-py3-none-any.whl", hash = "sha256:09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759"},
     {file = "packaging-24.2.tar.gz", hash = "sha256:c228a6dc5e932d346bc5739379109d49e8853dd8223571c7c5b55260edc0b97f"},
@@ -1188,6 +1253,7 @@ version = "0.5.6"
 description = "Divides large result sets into pages for easier browsing"
 optional = false
 python-versions = "*"
+groups = ["docs"]
 files = [
     {file = "paginate-0.5.6.tar.gz", hash = "sha256:5e6007b6a9398177a7e1648d04fdd9f8c9766a1a945bceac82f1929e8c78af2d"},
 ]
@@ -1198,6 +1264,7 @@ version = "0.11.1"
 description = "Utility library for gitignore style pattern matching of file paths."
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "pathspec-0.11.1-py3-none-any.whl", hash = "sha256:d8af70af76652554bd134c22b3e8a1cc46ed7d91edcdd721ef1a0c51a84a5293"},
     {file = "pathspec-0.11.1.tar.gz", hash = "sha256:2798de800fa92780e33acca925945e9a19a133b715067cf165b8866c15a31687"},
@@ -1209,6 +1276,7 @@ version = "0.14.1"
 description = "Check PEP-8 naming conventions, plugin for flake8"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "pep8-naming-0.14.1.tar.gz", hash = "sha256:1ef228ae80875557eb6c1549deafed4dabbf3261cfcafa12f773fe0db9be8a36"},
     {file = "pep8_naming-0.14.1-py3-none-any.whl", hash = "sha256:63f514fc777d715f935faf185dedd679ab99526a7f2f503abb61587877f7b1c5"},
@@ -1223,6 +1291,7 @@ version = "3.9.1"
 description = "A small Python package for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
 optional = false
 python-versions = ">=3.7"
+groups = ["dev", "docs"]
 files = [
     {file = "platformdirs-3.9.1-py3-none-any.whl", hash = "sha256:ad8291ae0ae5072f66c16945166cb11c63394c7a3ad1b1bc9828ca3162da8c2f"},
     {file = "platformdirs-3.9.1.tar.gz", hash = "sha256:1b42b450ad933e981d56e59f1b97495428c9bd60698baab9f3eb3d00d5822421"},
@@ -1238,6 +1307,7 @@ version = "1.5.0"
 description = "plugin and hook calling mechanisms for python"
 optional = false
 python-versions = ">=3.8"
+groups = ["test"]
 files = [
     {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"},
     {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"},
@@ -1253,6 +1323,7 @@ version = "4.1.0"
 description = "A framework for managing and maintaining multi-language pre-commit hooks."
 optional = false
 python-versions = ">=3.9"
+groups = ["dev"]
 files = [
     {file = "pre_commit-4.1.0-py2.py3-none-any.whl", hash = "sha256:d29e7cb346295bcc1cc75fc3e92e343495e3ea0196c9ec6ba53f49f10ab6ae7b"},
     {file = "pre_commit-4.1.0.tar.gz", hash = "sha256:ae3f018575a588e30dfddfab9a05448bfbd6b73d78709617b5a2b853549716d4"},
@@ -1269,8 +1340,10 @@ virtualenv = ">=20.10.0"
 name = "py4j"
 version = "0.10.9.7"
 description = "Enables Python programs to dynamically access arbitrary Java objects"
-optional = false
+optional = true
 python-versions = "*"
+groups = ["main"]
+markers = "extra == \"pyspark\""
 files = [
     {file = "py4j-0.10.9.7-py2.py3-none-any.whl", hash = "sha256:85defdfd2b2376eb3abf5ca6474b51ab7e0de341c75a02f46dc9b5976f5a5c1b"},
     {file = "py4j-0.10.9.7.tar.gz", hash = "sha256:0b6e5315bb3ada5cf62ac651d107bb2ebc02def3dee9d9548e3baac644ea8dbb"},
@@ -1282,6 +1355,7 @@ version = "2.12.0"
 description = "Python style guide checker"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "pycodestyle-2.12.0-py2.py3-none-any.whl", hash = "sha256:949a39f6b86c3e1515ba1787c2022131d165a8ad271b11370a8819aa070269e4"},
     {file = "pycodestyle-2.12.0.tar.gz", hash = "sha256:442f950141b4f43df752dd303511ffded3a04c2b6fb7f65980574f0c31e6e79c"},
@@ -1293,6 +1367,8 @@ version = "2.21"
 description = "C parser in Python"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+groups = ["dev"]
+markers = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and sys_platform == \"linux\" and platform_python_implementation != \"PyPy\""
 files = [
     {file = "pycparser-2.21-py2.py3-none-any.whl", hash = "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9"},
     {file = "pycparser-2.21.tar.gz", hash = "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"},
@@ -1304,6 +1380,7 @@ version = "2.10.6"
 description = "Data validation using Python type hints"
 optional = false
 python-versions = ">=3.8"
+groups = ["main"]
 files = [
     {file = "pydantic-2.10.6-py3-none-any.whl", hash = "sha256:427d664bf0b8a2b34ff5dd0f5a18df00591adcee7198fbd71981054cef37b584"},
     {file = "pydantic-2.10.6.tar.gz", hash = "sha256:ca5daa827cce33de7a42be142548b0096bf05a7e7b365aebfa5f8eeec7128236"},
@@ -1324,6 +1401,7 @@ version = "2.27.2"
 description = "Core functionality for Pydantic validation and serialization"
 optional = false
 python-versions = ">=3.8"
+groups = ["main"]
 files = [
     {file = "pydantic_core-2.27.2-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:2d367ca20b2f14095a8f4fa1210f5a7b78b8a20009ecced6b12818f455b1e9fa"},
     {file = "pydantic_core-2.27.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:491a2b73db93fab69731eaee494f320faa4e093dbed776be1a829c2eb222c34c"},
@@ -1436,6 +1514,7 @@ version = "3.2.0"
 description = "passive checker of Python programs"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "pyflakes-3.2.0-py2.py3-none-any.whl", hash = "sha256:84b5be138a2dfbb40689ca07e2152deb896a65c3a3e24c251c5c62489568074a"},
     {file = "pyflakes-3.2.0.tar.gz", hash = "sha256:1c61603ff154621fb2a9172037d84dca3500def8c8b630657d1701f026f8af3f"},
@@ -1447,6 +1526,7 @@ version = "2.16.1"
 description = "Pygments is a syntax highlighting package written in Python."
 optional = false
 python-versions = ">=3.7"
+groups = ["dev", "docs"]
 files = [
     {file = "Pygments-2.16.1-py3-none-any.whl", hash = "sha256:13fc09fa63bc8d8671a6d247e1eb303c4b343eaee81d861f3404db2935653692"},
     {file = "Pygments-2.16.1.tar.gz", hash = "sha256:1daff0494820c69bc8941e407aa20f577374ee88364ee10a98fdbe0aece96e29"},
@@ -1461,6 +1541,7 @@ version = "10.14.3"
 description = "Extension pack for Python Markdown."
 optional = false
 python-versions = ">=3.8"
+groups = ["docs"]
 files = [
     {file = "pymdown_extensions-10.14.3-py3-none-any.whl", hash = "sha256:05e0bee73d64b9c71a4ae17c72abc2f700e8bc8403755a00580b49a4e9f189e9"},
     {file = "pymdown_extensions-10.14.3.tar.gz", hash = "sha256:41e576ce3f5d650be59e900e4ceff231e0aed2a88cf30acaee41e02f063a061b"},
@@ -1479,6 +1560,7 @@ version = "3.2.1"
 description = "pyparsing module - Classes and methods to define and execute parsing grammars"
 optional = false
 python-versions = ">=3.9"
+groups = ["main"]
 files = [
     {file = "pyparsing-3.2.1-py3-none-any.whl", hash = "sha256:506ff4f4386c4cec0590ec19e6302d3aedb992fdc02c761e90416f158dacf8e1"},
     {file = "pyparsing-3.2.1.tar.gz", hash = "sha256:61980854fd66de3a90028d679a954d5f2623e83144b5afe5ee86f43d762e5f0a"},
@@ -1491,8 +1573,10 @@ diagrams = ["jinja2", "railroad-diagrams"]
 name = "pyspark"
 version = "3.5.4"
 description = "Apache Spark Python API"
-optional = false
+optional = true
 python-versions = ">=3.8"
+groups = ["main"]
+markers = "extra == \"pyspark\""
 files = [
     {file = "pyspark-3.5.4.tar.gz", hash = "sha256:1c2926d63020902163f58222466adf6f8016f6c43c1f319b8e7a71dbaa05fc51"},
 ]
@@ -1513,6 +1597,7 @@ version = "8.3.4"
 description = "pytest: simple powerful testing with Python"
 optional = false
 python-versions = ">=3.8"
+groups = ["test"]
 files = [
     {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"},
     {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"},
@@ -1535,6 +1620,7 @@ version = "6.0.0"
 description = "Pytest plugin for measuring coverage."
 optional = false
 python-versions = ">=3.9"
+groups = ["test"]
 files = [
     {file = "pytest-cov-6.0.0.tar.gz", hash = "sha256:fde0b595ca248bb8e2d76f020b465f3b107c9632e6a1d1705f17834c89dcadc0"},
     {file = "pytest_cov-6.0.0-py3-none-any.whl", hash = "sha256:eee6f1b9e61008bd34975a4d5bab25801eb31898b032dd55addc93e96fcaaa35"},
@@ -1553,6 +1639,7 @@ version = "0.5.2"
 description = "A py.test plugin that parses environment files before running tests"
 optional = false
 python-versions = "*"
+groups = ["test"]
 files = [
     {file = "pytest-dotenv-0.5.2.tar.gz", hash = "sha256:2dc6c3ac6d8764c71c6d2804e902d0ff810fa19692e95fe138aefc9b1aa73732"},
     {file = "pytest_dotenv-0.5.2-py3-none-any.whl", hash = "sha256:40a2cece120a213898afaa5407673f6bd924b1fa7eafce6bda0e8abffe2f710f"},
@@ -1568,6 +1655,7 @@ version = "2.8.2"
 description = "Extensions to the standard Python datetime module"
 optional = false
 python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7"
+groups = ["docs"]
 files = [
     {file = "python-dateutil-2.8.2.tar.gz", hash = "sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86"},
     {file = "python_dateutil-2.8.2-py2.py3-none-any.whl", hash = "sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9"},
@@ -1582,6 +1670,7 @@ version = "1.0.0"
 description = "Read key-value pairs from a .env file and set them as environment variables"
 optional = false
 python-versions = ">=3.8"
+groups = ["test"]
 files = [
     {file = "python-dotenv-1.0.0.tar.gz", hash = "sha256:a8df96034aae6d2d50a4ebe8216326c61c3eb64836776504fcca410e5937a3ba"},
     {file = "python_dotenv-1.0.0-py3-none-any.whl", hash = "sha256:f5971a9226b701070a4bf2c38c89e5a3f0d64de8debda981d1db98583009122a"},
@@ -1596,6 +1685,8 @@ version = "0.2.2"
 description = "A (partial) reimplementation of pywin32 using ctypes/cffi"
 optional = false
 python-versions = ">=3.6"
+groups = ["dev"]
+markers = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and sys_platform == \"win32\""
 files = [
     {file = "pywin32-ctypes-0.2.2.tar.gz", hash = "sha256:3426e063bdd5fd4df74a14fa3cf80a0b42845a87e1d1e81f6549f9daec593a60"},
     {file = "pywin32_ctypes-0.2.2-py3-none-any.whl", hash = "sha256:bf490a1a709baf35d688fe0ecf980ed4de11d2b3e37b51e5442587a75d9957e7"},
@@ -1607,6 +1698,7 @@ version = "6.0.1"
 description = "YAML parser and emitter for Python"
 optional = false
 python-versions = ">=3.6"
+groups = ["dev", "docs"]
 files = [
     {file = "PyYAML-6.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a"},
     {file = "PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f"},
@@ -1667,6 +1759,7 @@ version = "0.1"
 description = "A custom YAML tag for referencing environment variables in YAML files. "
 optional = false
 python-versions = ">=3.6"
+groups = ["docs"]
 files = [
     {file = "pyyaml_env_tag-0.1-py3-none-any.whl", hash = "sha256:af31106dec8a4d68c60207c1886031cbf839b68aa7abccdb19868200532c2069"},
     {file = "pyyaml_env_tag-0.1.tar.gz", hash = "sha256:70092675bda14fdec33b31ba77e7543de9ddc88f2e5b99160396572d11525bdb"},
@@ -1681,6 +1774,7 @@ version = "40.0"
 description = "readme_renderer is a library for rendering \"readme\" descriptions for Warehouse"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "readme_renderer-40.0-py3-none-any.whl", hash = "sha256:e18feb2a1e7706f2865b81ebb460056d93fb29d69daa10b223c00faa7bd9a00a"},
     {file = "readme_renderer-40.0.tar.gz", hash = "sha256:9f77b519d96d03d7d7dce44977ba543090a14397c4f60de5b6eb5b8048110aa4"},
@@ -1700,6 +1794,7 @@ version = "2022.10.31"
 description = "Alternative regular expression module, to replace re."
 optional = false
 python-versions = ">=3.6"
+groups = ["docs"]
 files = [
     {file = "regex-2022.10.31-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:a8ff454ef0bb061e37df03557afda9d785c905dab15584860f982e88be73015f"},
     {file = "regex-2022.10.31-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:1eba476b1b242620c266edf6325b443a2e22b633217a9835a52d8da2b5c051f9"},
@@ -1797,6 +1892,7 @@ version = "2.32.0"
 description = "Python HTTP for Humans."
 optional = false
 python-versions = ">=3.8"
+groups = ["dev", "docs"]
 files = [
     {file = "requests-2.32.0-py3-none-any.whl", hash = "sha256:f2c3881dddb70d056c5bd7600a4fae312b2a300e39be6a118d30b90bd27262b5"},
     {file = "requests-2.32.0.tar.gz", hash = "sha256:fa5490319474c82ef1d2c9bc459d3652e3ae4ef4c4ebdd18a21145a47ca4b6b8"},
@@ -1818,6 +1914,7 @@ version = "1.0.0"
 description = "A utility belt for advanced users of python-requests"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
+groups = ["dev"]
 files = [
     {file = "requests-toolbelt-1.0.0.tar.gz", hash = "sha256:7681a0a3d047012b5bdc0ee37d7f8f07ebe76ab08caeccfc3921ce23c88d5bc6"},
     {file = "requests_toolbelt-1.0.0-py2.py3-none-any.whl", hash = "sha256:cccfdd665f0a24fcf4726e690f65639d272bb0637b9b92dfd91a5568ccf6bd06"},
@@ -1832,6 +1929,7 @@ version = "2.0.0"
 description = "Validating URI References per RFC 3986"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
 files = [
     {file = "rfc3986-2.0.0-py2.py3-none-any.whl", hash = "sha256:50b1502b60e289cb37883f3dfd34532b8873c7de9f49bb546641ce9cbd256ebd"},
     {file = "rfc3986-2.0.0.tar.gz", hash = "sha256:97aacf9dbd4bfd829baad6e6309fa6573aaf1be3f6fa735c8ab05e46cecb261c"},
@@ -1846,6 +1944,7 @@ version = "13.4.2"
 description = "Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal"
 optional = false
 python-versions = ">=3.7.0"
+groups = ["dev"]
 files = [
     {file = "rich-13.4.2-py3-none-any.whl", hash = "sha256:8f87bc7ee54675732fa66a05ebfe489e27264caeeff3728c945d25971b6485ec"},
     {file = "rich-13.4.2.tar.gz", hash = "sha256:d653d6bccede5844304c605d5aac802c7cf9621efd700b46c7ec2b51ea914898"},
@@ -1864,6 +1963,8 @@ version = "3.3.3"
 description = "Python bindings to FreeDesktop.org Secret Service API"
 optional = false
 python-versions = ">=3.6"
+groups = ["dev"]
+markers = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and sys_platform == \"linux\""
 files = [
     {file = "SecretStorage-3.3.3-py3-none-any.whl", hash = "sha256:f356e6628222568e3af06f2eba8df495efa13b3b63081dafd4f7d9a7b7bc9f99"},
     {file = "SecretStorage-3.3.3.tar.gz", hash = "sha256:2403533ef369eca6d2ba81718576c5e0f564d5cca1b58f73a8b23e7d4eeebd77"},
@@ -1879,6 +1980,7 @@ version = "75.8.0"
 description = "Easily download, build, install, upgrade, and uninstall Python packages"
 optional = false
 python-versions = ">=3.9"
+groups = ["dev"]
 files = [
     {file = "setuptools-75.8.0-py3-none-any.whl", hash = "sha256:e3982f444617239225d675215d51f6ba05f845d4eec313da4418fdbb56fb27e3"},
     {file = "setuptools-75.8.0.tar.gz", hash = "sha256:c5afc8f407c626b8313a86e10311dd3f661c6cd9c09d4bf8c15c0e11f9f2b0e6"},
@@ -1899,6 +2001,7 @@ version = "1.16.0"
 description = "Python 2 and 3 compatibility utilities"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"
+groups = ["dev", "docs"]
 files = [
     {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"},
     {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"},
@@ -1910,6 +2013,7 @@ version = "2.4.0"
 description = "Sorted Containers -- Sorted List, Sorted Dict, Sorted Set"
 optional = false
 python-versions = "*"
+groups = ["test"]
 files = [
     {file = "sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"},
     {file = "sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88"},
@@ -1921,10 +2025,12 @@ version = "2.0.1"
 description = "A lil' TOML parser"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev", "test"]
 files = [
     {file = "tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc"},
     {file = "tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"},
 ]
+markers = {dev = "python_version < \"3.11\"", test = "python_full_version <= \"3.11.0a6\""}
 
 [[package]]
 name = "twine"
@@ -1932,6 +2038,7 @@ version = "6.1.0"
 description = "Collection of utilities for publishing packages on PyPI"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev"]
 files = [
     {file = "twine-6.1.0-py3-none-any.whl", hash = "sha256:a47f973caf122930bf0fbbf17f80b83bc1602c9ce393c7845f289a3001dc5384"},
     {file = "twine-6.1.0.tar.gz", hash = "sha256:be324f6272eff91d07ee93f251edf232fc647935dd585ac003539b42404a8dbd"},
@@ -1958,10 +2065,12 @@ version = "4.12.2"
 description = "Backported and Experimental Type Hints for Python 3.8+"
 optional = false
 python-versions = ">=3.8"
+groups = ["main", "dev", "docs"]
 files = [
     {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"},
     {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"},
 ]
+markers = {docs = "python_version < \"3.10\""}
 
 [[package]]
 name = "tzdata"
@@ -1969,6 +2078,7 @@ version = "2025.1"
 description = "Provider of IANA time zone data"
 optional = false
 python-versions = ">=2"
+groups = ["test"]
 files = [
     {file = "tzdata-2025.1-py2.py3-none-any.whl", hash = "sha256:7e127113816800496f027041c570f50bcd464a020098a3b6b199517772303639"},
     {file = "tzdata-2025.1.tar.gz", hash = "sha256:24894909e88cdb28bd1636c6887801df64cb485bd593f2fd83ef29075a81d694"},
@@ -1980,6 +2090,7 @@ version = "2.2.2"
 description = "HTTP library with thread-safe connection pooling, file post, and more."
 optional = false
 python-versions = ">=3.8"
+groups = ["dev", "docs"]
 files = [
     {file = "urllib3-2.2.2-py3-none-any.whl", hash = "sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472"},
     {file = "urllib3-2.2.2.tar.gz", hash = "sha256:dd505485549a7a552833da5e6063639d0d177c04f23bc3864e41e5dc5f612168"},
@@ -1997,6 +2108,7 @@ version = "20.26.6"
 description = "Virtual Python Environment builder"
 optional = false
 python-versions = ">=3.7"
+groups = ["dev"]
 files = [
     {file = "virtualenv-20.26.6-py3-none-any.whl", hash = "sha256:7345cc5b25405607a624d8418154577459c3e0277f5466dd79c49d5e492995f2"},
     {file = "virtualenv-20.26.6.tar.gz", hash = "sha256:280aede09a2a5c317e409a00102e7077c6432c5a38f0ef938e643805a7ad2c48"},
@@ -2017,6 +2129,7 @@ version = "3.0.0"
 description = "Filesystem events monitoring"
 optional = false
 python-versions = ">=3.7"
+groups = ["docs"]
 files = [
     {file = "watchdog-3.0.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:336adfc6f5cc4e037d52db31194f7581ff744b67382eb6021c868322e32eef41"},
     {file = "watchdog-3.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:a70a8dcde91be523c35b2bf96196edc5730edb347e374c7de7cd20c43ed95397"},
@@ -2056,6 +2169,7 @@ version = "0.5.1"
 description = "Character encoding aliases for legacy web content"
 optional = false
 python-versions = "*"
+groups = ["dev"]
 files = [
     {file = "webencodings-0.5.1-py2.py3-none-any.whl", hash = "sha256:a0af1213f3c2226497a97e2b3aa01a7e4bee4f403f95be16fc9acd2947514a78"},
     {file = "webencodings-0.5.1.tar.gz", hash = "sha256:b36a1c245f2d304965eb4e0a82848379241dc04b865afcc4aab16748587e1923"},
@@ -2067,16 +2181,21 @@ version = "3.19.1"
 description = "Backport of pathlib-compatible object wrapper for zip files"
 optional = false
 python-versions = ">=3.8"
+groups = ["dev", "docs"]
 files = [
     {file = "zipp-3.19.1-py3-none-any.whl", hash = "sha256:2828e64edb5386ea6a52e7ba7cdb17bb30a73a858f5eb6eb93d8d36f5ea26091"},
     {file = "zipp-3.19.1.tar.gz", hash = "sha256:35427f6d5594f4acf82d25541438348c26736fa9b3afa2754bcd63cdb99d8e8f"},
 ]
+markers = {dev = "(platform_machine != \"ppc64le\" and platform_machine != \"s390x\") and python_version < \"3.12\" or python_version < \"3.10\"", docs = "python_version < \"3.10\""}
 
 [package.extras]
 doc = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-lint"]
 test = ["big-O", "jaraco.functools", "jaraco.itertools", "jaraco.test", "more-itertools", "pytest (>=6,!=8.1.*)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=2.2)", "pytest-ignore-flaky", "pytest-mypy", "pytest-ruff (>=0.2.1)"]
 
+[extras]
+pyspark = ["pyspark"]
+
 [metadata]
-lock-version = "2.0"
+lock-version = "2.1"
 python-versions = "^3.9,<4.0.0"
-content-hash = "565d7989d437fb72d8c68021deb05d2e6ae612aae526cc5a7f4b2aeb003621fe"
+content-hash = "9e5e0a7c0bdeda155e438d86d601ede687a2ee1295eb49a2a19625f53c709b2c"
diff --git a/pyproject.toml b/pyproject.toml
index 5318a71..3d572a5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -9,10 +9,13 @@ packages = [{include = "sparkdantic", from = "src"}]
 [tool.poetry.dependencies]
 python = "^3.9,<4.0.0"
 pydantic = "^2.1.1"
-pyspark = "^3.3.0"
+pyspark = { version = "^3.3.0", optional = true }
 jmespath = "^1.0.1"
 pyparsing = "^3.1.1"
+packaging = "^24.2"
 
+[tool.poetry.extras]
+pyspark = ["pyspark"]
 
 [tool.poetry.group.dev.dependencies]
 bumpversion = "^0.6.0"
diff --git a/src/sparkdantic/__init__.py b/src/sparkdantic/__init__.py
index 002280a..3b38faf 100644
--- a/src/sparkdantic/__init__.py
+++ b/src/sparkdantic/__init__.py
@@ -2,10 +2,11 @@
 __author__ = 'Mitchell Lisle'
 __email__ = 'm.lisle90@gmail.com'
 
-from sparkdantic.model import SparkField, SparkModel, create_spark_schema
+from sparkdantic.model import SparkField, SparkModel, create_json_spark_schema, create_spark_schema
 
 __all__ = [
     'SparkField',
     'SparkModel',
     'create_spark_schema',
+    'create_json_spark_schema',
 ]
diff --git a/src/sparkdantic/exceptions.py b/src/sparkdantic/exceptions.py
index 74e2441..1ed9da7 100644
--- a/src/sparkdantic/exceptions.py
+++ b/src/sparkdantic/exceptions.py
@@ -1,2 +1,6 @@
 class TypeConversionError(Exception):
     """Error converting a model field type to a PySpark type"""
+
+
+class SparkdanticImportError(ImportError):
+    """Error importing PySpark. Raised when PySpark is not installed or the installed version is not supported."""
diff --git a/src/sparkdantic/model.py b/src/sparkdantic/model.py
index ae7df01..9d77802 100644
--- a/src/sparkdantic/model.py
+++ b/src/sparkdantic/model.py
@@ -5,16 +5,34 @@
 from decimal import Decimal
 from enum import Enum
 from types import MappingProxyType
-from typing import Annotated, Any, Literal, Optional, Type, Union, get_args, get_origin
+from typing import (
+    TYPE_CHECKING,
+    Annotated,
+    Any,
+    Dict,
+    Literal,
+    Optional,
+    Type,
+    Union,
+    get_args,
+    get_origin,
+)
 from uuid import UUID
 
 from pydantic import AliasChoices, AliasPath, BaseModel, ConfigDict, Field, SecretBytes, SecretStr
 from pydantic.fields import FieldInfo
 from pydantic.json_schema import JsonSchemaMode
-from pyspark.sql import types as spark_types
 
+from sparkdantic import utils
 from sparkdantic.exceptions import TypeConversionError
 
+if utils.have_pyspark:
+    utils.require_pyspark_version_in_range()
+    from pyspark.sql.types import DataType, StructType
+
+if TYPE_CHECKING:
+    from pyspark.sql.types import DataType, StructType
+
 if sys.version_info > (3, 10):
     from types import UnionType  # pragma: no cover
 else:
@@ -31,25 +49,25 @@
 
 _type_mapping = MappingProxyType(
     {
-        int: spark_types.IntegerType,
-        float: spark_types.DoubleType,
-        str: spark_types.StringType,
-        SecretStr: spark_types.StringType,
-        bool: spark_types.BooleanType,
-        bytes: spark_types.BinaryType,
-        SecretBytes: spark_types.BinaryType,
-        list: spark_types.ArrayType,
-        dict: spark_types.MapType,
-        datetime: spark_types.TimestampType,
-        date: spark_types.DateType,
-        Decimal: spark_types.DecimalType,
-        timedelta: spark_types.DayTimeIntervalType,
-        UUID: spark_types.StringType,
+        int: 'integer',
+        float: 'double',
+        str: 'string',
+        SecretStr: 'string',
+        bool: 'boolean',
+        bytes: 'binary',
+        SecretBytes: 'binary',
+        list: 'array',
+        dict: 'map',
+        datetime: 'timestamp',
+        date: 'date',
+        Decimal: 'decimal',
+        timedelta: 'interval day to second',
+        UUID: 'string',
     }
 )
 
 
-def SparkField(*args, spark_type: Optional[Type[spark_types.DataType]] = None, **kwargs) -> Field:
+def SparkField(*args, spark_type: Optional[Union[Type['DataType'], str]] = None, **kwargs) -> Field:
     if spark_type is not None:
         kwargs['spark_type'] = spark_type
     return Field(*args, **kwargs)
@@ -70,7 +88,7 @@ def model_spark_schema(
         safe_casting: bool = False,
         by_alias: bool = True,
         mode: JsonSchemaMode = 'validation',
-    ) -> spark_types.StructType:
+    ) -> 'StructType':
         """Generates a PySpark schema from the model fields. This operates similarly to
         `pydantic.BaseModel.model_json_schema()`.
 
@@ -84,14 +102,34 @@ def model_spark_schema(
         """
         return create_spark_schema(cls, safe_casting, by_alias, mode)
 
+    @classmethod
+    def model_json_spark_schema(
+        cls,
+        safe_casting: bool = False,
+        by_alias: bool = True,
+        mode: JsonSchemaMode = 'validation',
+    ) -> Dict[str, Any]:
+        """Generates a PySpark JSON compatible schema from the model fields. This operates similarly to
+        `pydantic.BaseModel.model_json_schema()`.
 
-def create_spark_schema(
+        Args:
+            safe_casting (bool): Indicates whether to use safe casting for integer types.
+            by_alias (bool): Indicates whether to use attribute aliases (`serialization_alias` or `alias`) or not.
+            mode (pydantic.json_schema.JsonSchemaMode): The mode in which to generate the schema.
+
+        Returns:
+            Dict[str, Any]: The generated PySpark JSON schema.
+        """
+        return create_json_spark_schema(cls, safe_casting, by_alias, mode)
+
+
+def create_json_spark_schema(
     model: Type[BaseModelOrSparkModel],
     safe_casting: bool = False,
     by_alias: bool = True,
     mode: JsonSchemaMode = 'validation',
-) -> spark_types.StructType:
-    """Generates a PySpark schema from the model fields. This operates similarly to
+) -> Dict[str, Any]:
+    """Generates a PySpark JSON compatible schema from the model fields. This operates similarly to
     `pydantic.BaseModel.model_json_schema()`.
 
     Args:
@@ -101,7 +139,7 @@ def create_spark_schema(
         mode (pydantic.json_schema.JsonSchemaMode): The mode in which to generate the schema.
 
     Returns:
-        pyspark.sql.types.StructType: The generated PySpark schema.
+        Dict[str, Any]: The generated PySpark JSON schema
     """
     if not _is_base_model(model):
         raise TypeError('`model` must be of type `SparkModel` or `pydantic.BaseModel`')
@@ -118,29 +156,70 @@ def create_spark_schema(
         override = field_info_extra.get('spark_type')
         field_type = _get_union_type_arg(info.annotation)
 
+        spark_type: Union[str, Dict[str, Any]]
+
         try:
             if _is_base_model(field_type):
-                spark_type = create_spark_schema(field_type, safe_casting, by_alias, mode)
+                spark_type = create_json_spark_schema(field_type, safe_casting, by_alias, mode)
             elif override is not None:
-                if not inspect.isclass(override) or not issubclass(override, spark_types.DataType):
-                    raise TypeError(
-                        '`spark_type` override should be a `pyspark.sql.types.DataType`'
-                    )
-                spark_type = override()
-            elif inspect.isclass(field_type) and issubclass(field_type, spark_types.DataType):
-                spark_type = field_type()
+                if isinstance(override, str):
+                    spark_type = override
+                elif utils.have_pyspark and _is_spark_datatype(override):
+                    spark_type = override.typeName()
+                else:
+                    msg = '`spark_type` override should be a `str` type name (e.g. long)'
+                    if utils.have_pyspark:
+                        msg += ' or `pyspark.sql.types.DataType` (e.g. LongType)'
+                    msg += f', but got {override}'
+                    raise TypeError(msg)
+            elif isinstance(field_type, str):
+                spark_type = field_type
+            elif utils.have_pyspark and _is_spark_datatype(field_type):
+                spark_type = field_type.typeName()
             else:
                 spark_type = _from_python_type(field_type, info.metadata, safe_casting)
-        except Exception as e:
-            raise TypeConversionError(f'Error converting field `{name}` to PySpark type') from e
+        except Exception as raised_error:
+            raise TypeConversionError(
+                f'Error converting field `{name}` to PySpark type'
+            ) from raised_error
 
         nullable = _is_optional(info.annotation)
-        struct_field = spark_types.StructField(name, spark_type, nullable)
+        struct_field = {
+            'name': name,
+            'type': spark_type,
+            'nullable': nullable,
+            'metadata': {},
+        }
         fields.append(struct_field)
-    return spark_types.StructType(fields)
+    return {
+        'type': 'struct',
+        'fields': fields,
+    }
+
+
+def create_spark_schema(
+    model: Type[BaseModelOrSparkModel],
+    safe_casting: bool = False,
+    by_alias: bool = True,
+    mode: JsonSchemaMode = 'validation',
+) -> 'StructType':
+    """Generates a PySpark schema from the model fields.
+
+    Args:
+        model (pydantic.BaseModel or SparkModel): The pydantic model to generate the schema from.
+        safe_casting (bool): Indicates whether to use safe casting for integer types.
+        by_alias (bool): Indicates whether to use attribute aliases (`serialization_alias` or `alias`) or not.
+        mode (pydantic.json_schema.JsonSchemaMode): The mode in which to generate the schema.
+
+    Returns:
+        pyspark.sql.types.StructType: The generated PySpark schema.
+    """
+    utils.require_pyspark()
+    json_schema = create_json_spark_schema(model, safe_casting, by_alias, mode)
+    return StructType.fromJson(json_schema)
 
 
-def _get_spark_type(t: Type) -> Type[spark_types.DataType]:
+def _get_spark_type(t: Type) -> str:
     """Returns the corresponding PySpark data type for a given Python type.
 
     Args:
@@ -182,7 +261,7 @@ def _from_python_type(
     type_: Type,
     metadata: list[Any],
     safe_casting: bool = False,
-) -> spark_types.DataType:
+) -> Union[str, Dict[str, Any]]:
     """Converts a Python type to a corresponding PySpark data type.
 
     Args:
@@ -191,12 +270,12 @@ def _from_python_type(
         safe_casting (bool): Indicates whether to use safe casting for integer types.
 
     Returns:
-        DataType: The corresponding PySpark data type.
+        Union[str, Dict[str, Any]]: The corresponding PySpark data type (dict for complex types).
     """
     py_type = _get_union_type_arg(type_)
 
     if _is_base_model(py_type):
-        return create_spark_schema(py_type, safe_casting)
+        return create_json_spark_schema(py_type, safe_casting)
 
     args = get_args(py_type)
     origin = get_origin(py_type)
@@ -208,13 +287,22 @@ def _from_python_type(
     if origin is list:
         element_type = _from_python_type(args[0], [])
         contains_null = _is_optional(args[0])
-        return spark_types.ArrayType(element_type, contains_null)
+        return {
+            'type': 'array',
+            'elementType': element_type,
+            'containsNull': contains_null,
+        }
 
     if origin is dict:
         key_type = _from_python_type(args[0], [])
         value_type = _from_python_type(args[1], [])
         value_contains_null = _is_optional(args[1])
-        return spark_types.MapType(key_type, value_type, value_contains_null)
+        return {
+            'type': 'map',
+            'keyType': key_type,
+            'valueType': value_type,
+            'valueContainsNull': value_contains_null,
+        }
 
     if origin is Literal:
         # PySpark doesn't have an equivalent type for Literal. To allow Literal usage with a model we check all the
@@ -235,16 +323,16 @@ def _from_python_type(
 
     spark_type = _get_spark_type(py_type)
 
-    if safe_casting is True and spark_type is spark_types.IntegerType:
-        spark_type = spark_types.LongType
+    if safe_casting is True and spark_type == 'integer':
+        spark_type = 'long'
 
-    if spark_type is spark_types.DecimalType:
+    if spark_type == 'decimal':
         meta = None if len(metadata) < 1 else deepcopy(metadata).pop()
         max_digits = getattr(meta, 'max_digits', 10)
         decimal_places = getattr(meta, 'decimal_places', 0)
-        return spark_types.DecimalType(precision=max_digits, scale=decimal_places)
+        return f'decimal({max_digits}, {decimal_places})'
 
-    return spark_type()
+    return spark_type
 
 
 def _is_base_model(cls: Type) -> bool:
@@ -310,3 +398,15 @@ def _get_field_alias(name: str, field_info: FieldInfo, mode: JsonSchemaMode = 'v
         else:
             alias = validation_alias
     return alias or name
+
+
+def _is_spark_datatype(t: Type) -> bool:
+    """Determines if a type is a PySpark DataType.
+
+    Args:
+        t (Type): The Python type to check.
+
+    Returns:
+        bool: a boolean indicating if the type is a PySpark DataType.
+    """
+    return inspect.isclass(t) and issubclass(t, DataType)
diff --git a/src/sparkdantic/utils.py b/src/sparkdantic/utils.py
new file mode 100644
index 0000000..2e0c92a
--- /dev/null
+++ b/src/sparkdantic/utils.py
@@ -0,0 +1,41 @@
+from packaging.version import Version
+
+from .exceptions import SparkdanticImportError
+
+MIN_PYSPARK_VERSION = '3.3.0'
+"""Inclusive minimum version of PySpark required"""
+MAX_PYSPARK_VERSION = '4.0.0'
+"""Exclusive maximum version of PySpark supported"""
+
+try:
+    import pyspark
+except ImportError as raised_error:
+    have_pyspark = False
+    pyspark_import_error = raised_error
+else:
+    have_pyspark = True
+    pyspark_import_error = None  # type: ignore
+
+
+def require_pyspark() -> None:
+    """
+    Raise SparkdanticImportError if PySpark is not installed
+    """
+    if not have_pyspark:
+        raise SparkdanticImportError(
+            'Pyspark is not installed. Install pyspark using `pip install sparkdantic[pyspark]`'
+        ) from pyspark_import_error
+
+
+def require_pyspark_version_in_range() -> None:
+    """
+    Raise SparkdanticImportError if an unsupported version of PySpark is installed
+    """
+
+    if not (
+        Version(MIN_PYSPARK_VERSION) <= Version(pyspark.__version__) < Version(MAX_PYSPARK_VERSION)
+    ):
+        raise SparkdanticImportError(
+            f'Pyspark version >={MIN_PYSPARK_VERSION},<{MAX_PYSPARK_VERSION} is required, '
+            f'but found {pyspark.__version__}'
+        )


Test patch:
diff --git a/tests/test_optional_pyspark_dependency.py b/tests/test_optional_pyspark_dependency.py
new file mode 100644
index 0000000..c3680e3
--- /dev/null
+++ b/tests/test_optional_pyspark_dependency.py
@@ -0,0 +1,69 @@
+import pydantic
+import pytest
+
+from sparkdantic import model, utils
+from sparkdantic.exceptions import SparkdanticImportError
+
+
+@pytest.fixture
+def no_pyspark(monkeypatch):
+    monkeypatch.setattr(utils, 'have_pyspark', False)
+
+
+def test_create_spark_schema_raises_import_error_when_no_pyspark(no_pyspark):
+    class NoPySparkModel(pydantic.BaseModel):
+        pass
+
+    with pytest.raises(SparkdanticImportError) as exc:
+        model.create_spark_schema(NoPySparkModel)
+    assert (
+        'Pyspark is not installed. Install pyspark using `pip install sparkdantic[pyspark]`'
+        == str(exc.value)
+    )
+
+
+def test_model_spark_schema_raises_import_error_when_no_pyspark(no_pyspark):
+    class NoPySparkModel(model.SparkModel):
+        pass
+
+    with pytest.raises(SparkdanticImportError) as exc:
+        NoPySparkModel.model_spark_schema()
+    assert (
+        'Pyspark is not installed. Install pyspark using `pip install sparkdantic[pyspark]`'
+        == str(exc.value)
+    )
+
+
+def test_no_pyspark_raises_import_error(no_pyspark):
+    with pytest.raises(SparkdanticImportError) as exc:
+        utils.require_pyspark()
+    assert (
+        'Pyspark is not installed. Install pyspark using `pip install sparkdantic[pyspark]`'
+        == str(exc.value)
+    )
+
+
+@pytest.mark.parametrize(
+    'version, raises_error',
+    [
+        ('2.4.0', True),
+        ('3.3.0', False),
+        ('3.5.0', False),
+        ('4.0.0', True),
+    ],
+)
+def test_require_pyspark_version_in_range(
+    version,
+    raises_error,
+    monkeypatch,
+):
+    monkeypatch.setattr(utils, 'have_pyspark', True)
+    monkeypatch.setattr(utils.pyspark, '__version__', version)
+
+    if raises_error:
+        with pytest.raises(SparkdanticImportError):
+            utils.require_pyspark_version_in_range()
+
+        assert f'Pyspark version >={utils.MIN_PYSPARK_VERSION},<{utils.MAX_PYSPARK_VERSION} is required, but found {version}'
+    else:
+        utils.require_pyspark_version_in_range()
diff --git a/tests/test_type_override.py b/tests/test_type_override.py
index 635ebb7..7a9e223 100644
--- a/tests/test_type_override.py
+++ b/tests/test_type_override.py
@@ -25,17 +25,33 @@ class MyModel(SparkModel):
     assert actual_schema == expected_schema
 
 
-def test_non_spark_type_override_raises_error():
-    class BadOverride(SparkModel):
-        id: UUID4 = Field(spark_type='a')
+def test_override_accepts_string_or_datatype():
+    class SupporrtedOverride(SparkModel):
+        s: int = Field(spark_type='string')
+        i: str = Field(spark_type=IntegerType)
+
+    expected_schema = StructType(
+        [
+            StructField('s', StringType(), False),
+            StructField('i', IntegerType(), False),
+        ]
+    )
+    actual_schema = SupporrtedOverride.model_spark_schema()
+    assert actual_schema == expected_schema
+
+
+def test_non_string_override_without_pyspark_raises_error():
+    class UnsupportedOverride(SparkModel):
+        id: str = Field(spark_type=int)
 
     with pytest.raises(TypeConversionError) as exc_info:
-        BadOverride.model_spark_schema()
+        UnsupportedOverride.model_spark_schema()
 
     assert 'Error converting field `id` to PySpark type' in str(exc_info.value)
     # Check cause
-    assert '`spark_type` override should be a `pyspark.sql.types.DataType`' in str(
-        exc_info.value.__context__
+    assert (
+        f'`spark_type` override should be a `str` type name (e.g. long) or `pyspark.sql.types.DataType` (e.g. LongType), but got {int}'
+        in str(exc_info.value.__cause__)
     )