Package: pyspark-stubs
A collection of the Apache Sparkstub
files. These
files were generated bystubgenand manually edited to include accurate type hints.Tests and configuration files have been originally contributed to theTypeshed project. Please refer
to itscontributors
listandlicensefor
details.ImportantThis projecthas been mergedwith the main Apache Spark repository (SPARK-32714). All further development for Spark 3.1 and onwards will be continued there.For Spark 2.4 and 3.0, development of this package will be continued, until their official deprecation.If your problem is specific to Spark 2.3 and 3.0 feel free to create an issue or open pull requests here.Otherwise, please checkthe official Spark JIRAandcontributing guidelines. If you create a JIRA ticket or Spark PR related to type hints, please ping me with[~zero323]or@zero323respectively. Thanks in advance.MotivationStatic error detection (seeSPARK-20631)Improved autocompletion.Installation and usagePlease note that the guidelines for distribution of type information is
still work in progress (PEP 561 - Distributing and Packaging Type
Information). Currently
installation script overlays existing Spark installations (pyistub
files are copied next to theirpycounterparts in the PySpark
installation directory). If this approach is not acceptable you can add stub
files to the search path manually.According toPEP
484:Third-party stub packages can use any location for stub storage.
Type checkers should search for them using PYTHONPATH.Moreover:Third-party stub packages can use any location for stub storage.
Type checkers should search for them using PYTHONPATH. A default
fallback directory that is always checked is
shared/typehints/python3.5/ (or 3.6, etc.)Please check usage before proceeding.The package is available onPYPI:pipinstallpyspark-stubsandconda-forge:condainstall-cconda-forgepyspark-stubsDepending on your environment you might also need a type checker, likeMypyorPytype[1], and autocompletion tool, likeJedi.EditorType checkingAutocompletionNotesAtom✔[2]✔[3]Through plugins.IPython/Jupyter Notebook✘[4]✔PyCharm✔✔PyDev✔[5]?VIM/Neovim✔[6]✔[7]Through plugins.Visual Studio Code✔[8]✔[9]Completion with pluginEnvironment independent / other editors✔[10]✔[11]ThroughMypyandJedi.This package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.PySpark Version CompatibilityPackage versions follow PySpark versions with exception to maintenance releases - i.e.pyspark-stubs==2.3.0should be compatible withpyspark>=2.3.0,<2.4.0.
Maintenance releases (post1,post2, …,postN) are reserved for internal annotations updates.API Coverage:As of release 2.4.0 most of the public API is covered. For details please checkAPI coverage document.See alsoSPARK-17333-Make pyspark interface friendly with static analysis.PySpark typing hintsandRevisiting PySpark type annotationsonApache Spark Developers List.DisclaimerApache Spark, Spark, PySpark, Apache, and the Spark logo aretrademarksofThe
Apache Software Foundation. This project is not owned, endorsed, or
sponsored by The Apache Software Foundation.Footnotes[1]Not supported or tested.[2]Requiresatom-mypyor equivalent.[3]Requiresautocomplete-python-jedior equivalent.[4]It is possibleto use magics to type check directly in the notebook. In general though, you’ll have to export whole notebook to.pyfile and run
type checker on the result.[5]Requires PyDev 7.0.3 or later.[6]TODO Usingvim-mypy,syntasticorNeomake.[7]Withjedi-vim.[8]WithMypy linter.[9]WithPython extension for Visual Studio Code.[10]Just use your favorite checker directly, optionally combined with tool likeentr.[11]SeeJedi editor plugins list.