Package: gelanis
gelanisGelanisis an enhanced version ofpysparkling.List of improvements:Data types of the resulting dataframes are equal to pysparkList of todos:Implemented since/until + be able to target a certain pyspark versionGet a drop-in API compatibility with pyspark (auto-injector is written, but more tests are needed here).Test the tests against both pyspark & pysparkling & compare the outputs so we’re 100% certain both libraries are API equal.Achieve API equality between pyspark & pysparkling (meaning that any public symbol should exist in both libraries).Increase tests to ensure 100% compatibility with pyspark.pysparklingPysparklingprovides a faster, more responsive way to develop programs
for PySpark. It enables code intended for Spark applications to execute
entirely in Python, without incurring the overhead of initializing and
passing data through the JVM and Hadoop. The focus is on having a lightweight
and fast implementation for small datasets at the expense of some data
resilience features and some parallel processing features.How does it work?To switch execution of a script from PySpark to pysparkling,
have the code initialize a pysparkling Context instead of a SparkContext, and
use the pysparkling Context to set up your RDDs. The beauty is you don’t have
to change a single line of code after the Context initialization, because
pysparkling’s API is (almost) exactly the same as PySpark’s. Since it’s so easy
to switch between PySpark and pysparkling, you can choose the right tool for your
use case.When would I use it?Say you are writing a Spark application because you
need robust computation on huge datasets, but you also want the same application
to provide fast answers on a small dataset. You’re finding Spark is not responsive
enough for your needs, but you don’t want to rewrite an entire separate application
for thesmall-answers-fastproblem. You’d rather reuse your Spark code but somehow
get it to run fast. Pysparkling bypasses the stuff that causes Spark’s long startup
times and less responsive feel.Here are a few areas where pysparkling excels:Small to medium-scale exploratory data analysisApplication prototypingLow-latency web deploymentsUnit testsInstallpipinstallpysparkling[s3,hdfs,streaming]Documentation:Other links:Github,,,FeaturesSupports URI schemess3://,hdfs://,gs://,http://andfile://for Amazon S3, HDFS, Google Storage, web and local file access.
Specify multiple files separated by comma.
Resolves*and?wildcards.Handles.gz,.zip,.lzma,.xz,.bz2,.tar,.tar.gzand.tar.bz2compressed files.
Supports reading of.7zfiles.Parallelization viamultiprocessing.Pool,concurrent.futures.ThreadPoolExecutoror any other Pool-like
objects that have amap(func, iterable)method.Plain pysparkling does not have any dependencies (usepip install pysparkling).
Some file access methods have optional dependencies:botofor AWS S3,requestsfor http,hdfsfor hdfsExamplesSome demos are in the notebooksdocs/demo.ipynbanddocs/iris.ipynb.Word CountfrompysparklingimportContextcounts=(Context().textFile('README.rst').map(lambdaline:''.join(chifch.isalnum()else' 'forchinline)).flatMap(lambdaline:line.split(' ')).map(lambdaword:(word,1)).reduceByKey(lambdaa,b:a+b))print(counts.collect())which prints a long list of pairs of words and their counts.