Repository: pyjanitor-devs/pyjanitor

Issue:
Require pyspark minimal version is v3.2.0 to cut duplicates codes
Since [pyspark v3.2.0](https://github.com/apache/spark/blob/5d45a415f3a29898d92380380cfd82bfc7f579ea/python/pyspark/pandas/extensions.py#L28-L64), it has contained `CachedAccessor`, `_register_accessor`, `_register_accessor`

janitor requires pyspark minimal version is v3.1.2 at present.
Compared to v3.1.2, v3.2.0 is a minor version.

https://github.com/pyjanitor-devs/pyjanitor/blob/4867b20273e99cb2f4c32278123d5c23c2ccaa67/janitor/spark/backend.py#L9-L37

Note: The pyspark in the [setup.py](https://github.com/pyjanitor-devs/pyjanitor/blob/4867b20273e99cb2f4c32278123d5c23c2ccaa67/.requirements/spark.txt#L9) file requires v3.2.1 but ci ([environment-dev.yml](https://github.com/pyjanitor-devs/pyjanitor/blob/4867b20273e99cb2f4c32278123d5c23c2ccaa67/environment-dev.yml#L41)) requires v3.1.2.


Discussion:
@Zeroto521 with PySpark 3.2.0 we'd be able to directly register our code on pyspark dataframes?
> @Zeroto521 with PySpark 3.2.0 we'd be able to directly register our code on pyspark dataframes?

Yes, it could easily import 'register_dataframe_accessor' from pyspark since 3.2.0 version.

But pyjanitor still needs to keep 'register_dataframe_method'.
That's such a cool find. Thank you so much, @Zeroto521 :heart:
@Zeroto521 I can't remember, but have we added you to the core dev team, btw? Having seen your contributions and patience with the project, I think you'd be a valuable addition to the team.
For more step, even `register_dataframe_method` could be simplified via `register_method_factory` in https://github.com/Zsailer/pandas_flavor/issues/23
> @Zeroto521 I can't remember, but have we added you to the core dev team, btw? 

No. But I'm happy to become the core maintainer of pyjanitor project.
Cool stuff! I'm going to add you onto the dev team. Could you ping me at http://www.shortwhale.com/ericmjl? I'll also add you onto the Discord chat that we use. (It's very low volume, btw, and it's more for social chat than dev chat.)
> Could you ping me at http://www.shortwhale.com/ericmjl?

Done.

> I'll also add you onto the Discord chat that we use. (It's very low volume, btw, and it's more for social chat than dev chat.)

The Discord service is blocked in China. So I'm not very frequently use it.
¯\_(ツ)_/¯

Resolving patch:
diff --git a/janitor/spark/backend.py b/janitor/spark/backend.py
--- a/janitor/spark/backend.py
+++ b/janitor/spark/backend.py
@@ -1,84 +1,20 @@
 """ Backend functions for pyspark."""
 
-import warnings
 from functools import wraps
 
-from janitor.utils import import_message
 
+try:
+    from pyspark.pandas.extensions import register_dataframe_accessor
 
-class CachedAccessor:
-    """
-    Custom property-like object (descriptor) for caching accessors.
-
-    Parameters
-    ----------
-    name : str
-        The namespace this will be accessed under, e.g. `df.foo`
-    accessor : cls
-        The class with the extension methods.
-
-    NOTE
-    ----
-    Modified based on pandas.core.accessor.
-    """
-
-    def __init__(self, name, accessor):
-        self._name = name
-        self._accessor = accessor
-
-    def __get__(self, obj, cls):
-        if obj is None:
-            # we're accessing the attribute of the class, i.e., Dataset.geo
-            return self._accessor
-        accessor_obj = self._accessor(obj)
-        # Replace the property with the accessor object. Inspired by:
-        # http://www.pydanny.com/cached-property.html
-        setattr(obj, self._name, accessor_obj)
-        return accessor_obj
-
-
-def _register_accessor(name, cls):
-    """
-    NOTE
-    ----
-    Modified based on pandas.core.accessor.
-    """
-
-    def decorator(accessor):
-        if hasattr(cls, name):
-            warnings.warn(
-                "registration of accessor {!r} under name {!r} for type "
-                "{!r} is overriding a preexisting attribute with the same "
-                "name.".format(accessor, name, cls),
-                UserWarning,
-                stacklevel=2,
-            )
-        setattr(cls, name, CachedAccessor(name, accessor))
-        return accessor
-
-    return decorator
-
-
-def register_dataframe_accessor(name):
-    """
-    NOTE
-    ----
-    Modified based on pandas.core.accessor.
-
-    .. # noqa: DAR101 name
-    .. # noqa: DAR201
-    """
-    try:
-        from pyspark.sql import DataFrame
-    except ImportError:
-        import_message(
-            submodule="spark",
-            package="pyspark",
-            conda_channel="conda-forge",
-            pip_install=True,
-        )
+except ImportError:
+    from janitor.utils import import_message
 
-    return _register_accessor(name, DataFrame)
+    import_message(
+        submodule="spark",
+        package="pyspark",
+        conda_channel="conda-forge",
+        pip_install=True,
+    )
 
 
 def register_dataframe_method(method):


Test patch: