Saravanan666/pypi-org huggingface dataset:
Package: pysparkling
pysparklingPysparklingprovides a faster, more responsive way to develop programs
for PySpark. It enables code intended for Spark applications to execute
entirely in Python, without incurring the overhead of initializing and
passing data through the JVM and Hadoop. The focus is on having a lightweight
and fast implementation for small datasets at the expense of some data
resilience features and some parallel processing features.How does it work?To switch execution of a script from PySpark to pysparkling,
have the code initialize a pysparkling Context instead of a SparkContext, and
use the pysparkling Context to set up your RDDs. The beauty is you don’t have
to change a single line of code after the Context initialization, because
pysparkling’s API is (almost) exactly the same as PySpark’s. Since it’s so easy
to switch between PySpark and pysparkling, you can choose the right tool for your
use case.When would I use it?Say you are writing a Spark application because you
need robust computation on huge datasets, but you also want the same application
to provide fast answers on a small dataset. You’re finding Spark is not responsive
enough for your needs, but you don’t want to rewrite an entire separate application
for thesmall-answers-fastproblem. You’d rather reuse your Spark code but somehow
get it to run fast. Pysparkling bypasses the stuff that causes Spark’s long startup
times and less responsive feel.Here are a few areas where pysparkling excels:Small to medium-scale exploratory data analysisApplication prototypingLow-latency web deploymentsUnit testsInstallpython3-mpipinstall"pysparkling[s3,hdfs,http,streaming]"Documentation:Other links:Github,,,FeaturesSupports URI schemess3://,hdfs://,gs://,http://andfile://for Amazon S3, HDFS, Google Storage, web and local file access.
Specify multiple files separated by comma.
Resolves*and?wildcards.Handles.gz,.zip,.lzma,.xz,.bz2,.tar,.tar.gzand.tar.bz2compressed files.
Supports reading of.7zfiles.Parallelization viamultiprocessing.Pool,concurrent.futures.ThreadPoolExecutoror any other Pool-like
objects that have amap(func, iterable)method.Plain pysparkling does not have any dependencies (usepip install pysparkling).
Some file access methods have optional dependencies:botofor AWS S3,requestsfor http,hdfsfor hdfsExamplesSome demos are in the notebooksdocs/demo.ipynbanddocs/iris.ipynb.Word CountfrompysparklingimportContextcounts=(Context().textFile('README.rst').map(lambdaline:''.join(chifch.isalnum()else' 'forchinline)).flatMap(lambdaline:line.split(' ')).map(lambdaword:(word,1)).reduceByKey(lambdaa,b:a+b))print(counts.collect())which prints a long list of pairs of words and their counts.
Package: pysparkplug
PySparkplug: Sparkplug B for PythonGetting StartedInstallationpysparkplugis a pip-installable packagehosted on PyPI. Getting started is as easy as:$pipinstallpysparkplugpysparkpluguses the Eclipse Paho™ MQTT Python Client, i.e.paho-mqtt, for low-level MQTT communication.UsageMore documentation to come later, but for now, you can find some example usage notebooks in thenotebooksdirectory.FeaturesFully type annotatedpysparkplug's various interfaces are fully type annotated, passingMypy's static type checker.
Package: pysparkplus
Pyspark extra functions ❤️Documentation🚀Features ✨️Wrapper Class!Simple use!Made with A.I. contribution 🤖Documentation and examples!!Development on go!References 🌍 🗄️Pyspark+Wise Up DataEmojisMaintainer 🤗 👨‍💻Sivio Liborio📧silvio.liborio@wiseupdata.comsilvio-de-melo-liborio
Package: pyspark-spy
pyspark-spyCollect and aggregate on spark events for profitz. In 🐍 way!Installationpipinstallpyspark-spyHow toYou register a listenerimportpyspark_spylistener=pyspark_spy.PersistingSparkListener()pyspark_spy.register_listener(spark_context,listener)Execute your spark job as usualspark_context.range(1,100).count()And you have all metrics collected!print(listener.stage_output_metrics_aggregate())OutputMetrics(bytesWritten=12861,recordsWritten=2426)Look Ma, no actions!Tested on python 3.5 - 3.7 and pyspark 2.3 - 3.0Available listenerspyspark_spy.interface.SparkListener- Base listener class.
It defineson_spark_event(event_name, java_event)method that you can implement yourself
for custom logic when any event is received.LoggingSparkListener- just logs event names received into supplied or automatically created logger.StdoutSparkListener- writes event names into stdoutPersistingSparkListener- saves spark events into internal bufferContextSparkListener- same as PersistingSparkListener but also allows you to record only events
occured within python context manager scope. More on that laterPersistingSparkListenerSpark events collected (as java objects):applicationEndapplicationStartblockManagerRemovedblockUpdatedenvironmentUpdateexecutorAddedexecutorMetricsUpdateexecutorRemovedjobEndjobStartotherEventstageCompletedstageSubmittedtaskEndtaskGettingResulttaskStartunpersistRDDlistener.java_events['executorMetricsUpdate']# -> List of py4j java objectsView all possible spark events and their fieldshttps://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/scheduler/SparkListener.htmlEvents converted to python objects:jobEndstageCompletedlistener.python_events['jobEnd']# -> List of java events converted to typed namedtuples.listener.jobEnd# sameAvailable aggregationsOnly inPersistingSparkListenerandContextSparkListenerstage_input_metrics_aggregate- sums up allstageCompletedevent inputMetrics into oneprint(listener.stage_input_metrics_aggregate())InputMetrics(bytesRead=21574,recordsRead=584)stage_output_metrics_aggregate- sums up allstageCompletedevent outputMetrics into oneprint(listener.stage_output_metrics_aggregate())OutputMetrics(bytesWritten=12861,recordsWritten=2426)ContextSparkListenerTo collect events from different actions and to build separate aggregations, useContextSparkListener.listener=ContextSparkListener()register_listener(sc,listener)withlistenerasevents:# events is basically another listenerrun_spark_job()events.stage_output_metrics_aggregate()# events collected only within context managerwithlistenerasevents_2:run_other_spark_job()events_2.stage_output_metrics_aggregate()# metrics collected during second joblistener.stage_output_metrics_aggregate()# metrics collected for all jobs
Package: pyspark-sugar
# pyspark-sugarSet python traceback on dataframe actions, enrich spark UI with actual business logic stages of spark application.## InstallationVia pip:```pip install pyspark-sugar```Directly from github:```pip install git+https://github.com/sashgorokhov/pyspark-sugar.git@master```## UsageConsider this synthetic example```pythonimport randomimport pysparkimport pyspark_sugarfrom pyspark.sql import functions as F# Set verbose job description through decorator@pyspark_sugar.job_description_decor('Get nulls after type casts')def get_incorrect_cast_cols(sdf, cols):"""Return columns with non-zero nulls amount across its values.:param cols: Subset of columns to check"""cols = set(cols) & set(sdf.columns)if not cols:return {}null_counts = sdf.agg(*[F.sum(F.isnull(col).cast('int')).alias(col) for col in cols]).collect()[0]return {col: null_counts[col] for col in cols if null_counts[col]}def main():sc = pyspark.SparkContext.getOrCreate()  # type: pyspark.SparkContextssc = pyspark.sql.SparkSession.builder.getOrCreate()# Define a job group. All actions inside job group will be grouped.with pyspark_sugar.job_group('ETL', None):sdf = ssc.createDataFrame([{'CONTACT_KEY': n, 'PRODUCT_KEY': random.choice(['1', '2', 'what', None])}for n in range(1000)]).repartition(2).cache()sdf = sdf.withColumn('PRODUCT_KEY', F.col('PRODUCT_KEY').cast('integer'))# Collect action inside this function will have nice and readable descriptionprint(get_incorrect_cast_cols(sdf, ['PRODUCT_KEY']))sdf = sdf.dropna(subset=['PRODUCT_KEY'])# You can define several job groups one after another.with pyspark_sugar.job_group('Analytics', 'Check dataframe metrics'):sdf.cache()# Set job description for actions made inside context manager.with pyspark_sugar.job_description('Get transactions count'):print(sdf.count())sdf.show()sc.stop()if __name__ == '__main__':# Patch almost-all dataframe actions with code that will set stack trace to job details descriptionwith pyspark_sugar.patch_dataframe_actions():main()```This how SparkUI pages will look like:![SparkUI jobs page screenshot](docs/screenshots/screenshot_jobs.png)![SparkUI sql page screenshot](docs/screenshots/screenshot_sql.png)Now SparkUI is full of human-readable descriptions so even you manager can read it! Sweet.
Package: pyspark-test
pyspark-testCheck that left and right spark DataFrame are equal.This function is intended to compare two spark DataFrames and output any differences. It is inspired from pandas testing module but for pyspark, and for use in unit tests. Additional parameters allow varying the strictness of the equality checks performed.Installationpip install pyspark-testUsageassert_pyspark_df_equal(left_df,actual_df)Additional Argumentscheck_dtype: To compare the data types of spark dataframe. Default truecheck_column_names: To compare column names. Default false. Not required of we are checking data types.check_columns_in_order: To check the columns should be in order or not. Default to falseorder_by: Column names with which dataframe must be sorted before comparing. Default None.ExampleimportdatetimefrompysparkimportSparkContextfrompyspark.sqlimportSparkSessionfrompyspark.sql.typesimport*frompyspark_testimportassert_pyspark_df_equalsc=SparkContext.getOrCreate(conf=conf)spark_session=SparkSession(sc)df_1=spark_session.createDataFrame(data=[[datetime.date(2020,1,1),'demo',1.123,10],[None,None,None,None],],schema=StructType([StructField('col_a',DateType(),True),StructField('col_b',StringType(),True),StructField('col_c',DoubleType(),True),StructField('col_d',LongType(),True),]),)df_2=spark_session.createDataFrame(data=[[datetime.date(2020,1,1),'demo',1.123,10],[None,None,None,None],],schema=StructType([StructField('col_a',DateType(),True),StructField('col_b',StringType(),True),StructField('col_c',DoubleType(),True),StructField('col_d',LongType(),True),]),)assert_pyspark_df_equal(df_1,df_2)
Package: pyspark-testing
pyspark-testing is testing framework for pysparkInstallationpyspark-testing is available at thePyPI# PyPI
$ pip install pyspark-testingBasic Usagefrom pyspark_testing import assert_dateframe_equal


def test_sample():
    data = [('sample', 1)]

    left = spark.createDataFrame(data)
    right = spark.createDataFrame(data)

    assert_dataframe_equal(left, right)LicenseMIT License (seeLICENSE).
Package: pyspark-tests
spark-testsA test framework, that defines several test doubles, to facilitate Python Spark application development.spark_tests.sql moduleDefines the following test doubles:FakeSparkSessionStubssql(sql_query)method to only log thesql_queries, not sending them to database for execution;table(table_mame)andcreateDataFrame(data[, schema, samplingRatio, verifySchema])methods delegate execution to the realSparkSession, but returns aFakeDataFrameinstead of aDataFrame;table(table_name)is often overridden in a subclass to return a table from a fake test database.FakeDataFramewritereturns aFakeDFWriter;Other methods work just like a realDataFrame, but returnFakeDataFrames instead ofDataFrames;FakeDFWriterStubs aDataFrameWriterto only logRows written, not writing them at all.spark_tests.delta moduleDefinesFakeDeltaTable, that stubsmerge(source, condition)to only log the merge operation, changing no data.spark_tests.datetime moduleDefines the following test doubles:FakeDatetimeStubsnow()method to return always a predefineddatetime.FakeDateStubstoday()method to return always a predefineddate.
Package: pyspark-types
PySpark Typespyspark_typesis a Python library that provides a simple way to map Python dataclasses to PySpark StructTypes.UsagePydanticPySparkBaseModel is a base class for PySpark models that provides methods for converting between PySpark Rows and Pydantic models.Here's an example of a Pydantic model that will be used to create a PySpark DataFrame:frompyspark_types.auxiliaryimportBoundDecimalfrompyspark_types.pydanticimportPySparkBaseModelclassPerson(PySparkBaseModel):name:strage:intaddresses:dict[str,str]salary:BoundDecimalTo create a PySpark DataFrame from a list of Person Pydantic models, we can use PySparkBaseModel.create_spark_dataframe() method.frompyspark.sqlimportSparkSessionspark=SparkSession.builder.appName("MyApp").getOrCreate()# create a list of Pydantic modelsdata=[Person(name="Alice",age=25,addresses={"home":"123 Main St","work":"456 Pine St"},salary=BoundDecimal("5000.00",precision=10,scale=2),),Person(name="Bob",age=30,addresses={"home":"789 Elm St","work":"321 Oak St"},salary=BoundDecimal("6000.50",precision=10,scale=2),),]# create a PySpark DataFrame from the list of Pydantic modelsdf=Person.create_spark_dataframe(data,spark)# show the contents of the DataFramedf.show()Output:+---+-----+--------------------+------+|age|name|addresses|salary|+---+-----+--------------------+------+|25|Alice|[home->123Main...|5000.00||30|Bob|[home->789Elm...|6000.50|+---+-----+--------------------+------+The PySparkBaseModel.create_spark_dataframe() method converts the list of Pydantic models to a list of dictionaries, and then creates a PySpark DataFrame from the list of dictionaries and schema generated from the Pydantic model.You can also generate a schema based on a Pydantic model by calling the PySparkBaseModel.schema() method:schema=PySparkBaseModel.schema(Person)This creates a PySpark schema for the Person Pydantic model.Note that if you have custom types, such as BoundDecimal, you will need to add support for them in PySparkBaseModel. For example, you can modify the PySparkBaseModel.dict() method to extract BoundDecimal values when mapping to DecimalType.DataclassesTo use pyspark_types, you first need to define a Python data class with the fields you want to map to PySpark. For example:fromdataclassesimportdataclass@dataclassclassPerson:name:strage:intis_student:boolTo map this data class to a PySpark StructType, you can use the map_dataclass_to_struct() function:frompyspark_typesimportmap_dataclass_to_structperson_struct=map_dataclass_to_struct(Person)This will return a PySpark StructType that corresponds to the Person data class.You can also use the apply_nullability() function to set the nullable flag for a given PySpark DataType:frompyspark.sql.typesimportStringTypefrompyspark_typesimportapply_nullabilitynullable_string_type=apply_nullability(StringType(),True)This will return a new PySpark StringType with the nullable flag set to True.
Package: pyspark-util
pyspark-utilA set of pyspark utility functions.importpyspark_utilaspsudata=[(1,2,3)]columns=['a','b','c']df=spark.createDataFrame(data,columns)prefixed=psu.prefix_columns(df,'x')prefixed.show()# output:+---+---+---+|x_a|x_b|x_c|+---+---+---+|1|2|3|+---+---+---+DevelopmentSetupdocker-compose build
docker-compose up -dLintdocker exec psu-cnt ./tools/lint.shTestdocker exec psu-cnt ./tools/test.sh
Package: pyspark-utils
The missing PySpark utils.UsageTo install:pip install pyspark-utils
# It also depends on absl-py.helperimport pyspark_utils.helper as spark_helper

# Nicely show rdd count and 3 items.
rdd = spark_helper.cache_and_log('MyRDD', rdd, 3)opimport pyspark_utils.op as spark_op

# RDD<key, value>  ->  RDD<new_key, value>
pair_rdd.map(spark_op.do_key(lambda key: new_key))

# RDD<key, value>  ->  RDD<result>
pair_rdd.map(spark_op.do_tuple(lambda key, value: result))

# RDD<key, value>  ->  RDD<value, key>
pair_rdd.map(spark_op.swap_kv())

# RDD<key, value>  ->  RDD<key, value> if func(key)
pair_rdd.filter(spark_op.filter_key(lambda key: true_or_false))

# RDD<key, value>  ->  RDD<key, value> if func(value)
pair_rdd.filter(spark_op.filter_value(lambda value: true_or_false))

# RDD<iteratable>  ->  RDD<tuple_or_list> with transformed values.
rdd.map(spark_op.do_tuple_elems(lambda elem: new_elem))
rdd.map(spark_op.do_list_elems(lambda elem: new_elem))

# RDD<path>  ->  RDD<path> if path matches any given fnmatch-style patterns
rdd.filter(spark_op.filter_path(['*.txt', '*.csv', 'path/a.???']))

# RDD<element>  ->  RDD<element, element>
rdd.keyBy(spark_op.identity)

# RDD<key, value>   ->   RDD<key, value> with keys in key_rdd
spark_op.filter_keys(pair_rdd, key_rdd)

# RDD<key, value>   ->   RDD<key, value> with keys in whitelist and not in blacklist
spark_op.filter_keys(pair_rdd, whitelist_key_rdd, blacklist_key_rdd)

# RDD<key, value>   ->   RDD<key, value> with keys not in key_rdd
spark_op.substract_keys(pair_rdd, key_rdd)

# RDD<element>   ->   RDD<element> where element is not None
rdd.filter(spark_op.not_none)

# RDD<key>   ->   RDD<key, value>
rdd.map(spark_op.value_by(lambda key: value))
Package: pyspark-val
pyspark-testPySpark validation & testing tooling.Installationpip install pyspark-valUsageassert_pyspark_df_equal(left_df,actual_df)Additional Argumentscheck_dtype: To compare the data types of spark dataframe. Default truecheck_column_names: To compare column names. Default false. Not required of we are checking data types.check_columns_in_order: To check the columns should be in order or not. Default to falseorder_by: Column names with which dataframe must be sorted before comparing. Default None.ExampleimportdatetimefrompysparkimportSparkContextfrompyspark.sqlimportSparkSessionfrompyspark.sql.typesimport*frompyspark_testimportassert_pyspark_df_equalsc=SparkContext.getOrCreate(conf=conf)spark_session=SparkSession(sc)df_1=spark_session.createDataFrame(data=[[datetime.date(2020,1,1),'demo',1.123,10],[None,None,None,None],],schema=StructType([StructField('col_a',DateType(),True),StructField('col_b',StringType(),True),StructField('col_c',DoubleType(),True),StructField('col_d',LongType(),True),]),)df_2=spark_session.createDataFrame(data=[[datetime.date(2020,1,1),'demo',1.123,10],[None,None,None,None],],schema=StructType([StructField('col_a',DateType(),True),StructField('col_b',StringType(),True),StructField('col_c',DoubleType(),True),StructField('col_d',LongType(),True),]),)assert_pyspark_df_equal(df_1,df_2)
Package: pyspark-vector-files
PySpark Vector FilesReadvector filesinto aSpark DataFramewith geometry encoded asWell Known Binary (WKB).Full documentation is availablehere.RequirementsThis library was developed usingDatabricks Runtime 10.4 LTSand uses the versions ofpython,pandasandpysparkthat come pre-installed on that runtime. However, it also requiresGDAL 3.4.3as this is the most recent version ofGDALavailable fromubuntugis-unstableas of 2022-08-11.You can installGDALon your cluster using aninit script. Seeherefor an example.Installpyspark-vector-filesWithin a Databricks notebook%pipinstallpyspark-vector-filesFrom the command linepython-mpipinstallpyspark-vector-filesQuick startRead the first layer from a file or files with given extension into a single Spark DataFrame:frompyspark_vector_filesimportread_vector_filessdf=read_vector_files(path="/path/to/files/",suffix=".ext",)More examples are availablehere.Local developmentTo ensure compatibility withDatabricks Runtime 10.4 LTS, this package was developed on a Linux machine running theUbuntu 20.04 LTSoperating system usingPython3.8.10,GDAL 3.4.3, andspark 3.2.1..InstallPython 3.8.10usingpyenvSee thepyenv-installer'sInstallation / Update / Uninstallationinstructions.Install Python 3.8.10 globally:pyenvinstall3.8.10Then install it locally in the repository you're using:pyenvlocal3.8.10InstallGDAL 3.4.3Add theUbuntuGIS unstable Private Package Archive (PPA)and update your package list:sudoadd-apt-repositoryppa:ubuntugis/ubuntugis-unstable\&&sudoapt-getupdateInstallgdal 3.4.3, I found I also had to install python3-gdal (even though
I'm going to use poetry to install it in a virtual environment later) to
avoid version conflicts:sudoapt-getinstall-ygdal-bin=3.4.3+dfsg-1~focal0\libgdal-dev=3.4.3+dfsg-1~focal0\python3-gdal=3.4.3+dfsg-1~focal0Verify the installation:ogrinfo--version# GDAL 3.4.3, released 2022/04/22Installpoetry 1.1.13See poetry'sosx / linux / bashonwindows install instructionsClone this repositorygitclonehttps://github.com/Defra-Data-Science-Centre-of-Excellence/pyspark_vector_files.gitInstall dependencies usingpoetrypoetryinstall
Package: pysparrow
SparrowSparrow (a combination ofSparkandarrow) is a Python mini library that enhances Spark with anarrow API.Intent is to make mappers and filters over RDD a bit more elegant and exciting. Author also feels that here developed API does have a more consitent feel.Consider and example of few operations on an RDD in native PySpark...rdd=spark.sparkContext.parallelize([(1,2.0,["a","b","c"]),(2,3.0,["b","c","d"]),(3,4.0,["c","d","e"]),(4,5.0,["d","e","f"]),(5,6.0,["e","f","g"]),])res=rdd.map(lambdax:x[2]).flatMap(lambdax:x).filter(lambdax:x=='b')and then on RDD extended with Sparrow:rdd=spark.sparkContext.parallelize([(1,2.0,["a","b","c"]),(2,3.0,["b","c","d"]),(3,4.0,["c","d","e"]),(4,5.0,["d","e","f"]),(5,6.0,["e","f","g"]),])res=(SparrowRDD(rdd)>>(lambdax:x[2])>>Flatten(lambdax:x)>>Filter(lambdax:x=='b'))
Package: pysparse
Pysparse is a fast sparse matrix library for Python. It provides several sparse
matrix storage formats and conversion methods. It also implements a number of
iterative solvers, preconditioners, and interfaces to efficient factorization
packages. Both low-level and high-level interfaces are available, each with
different strengths. PySparse is distributed under the FreeBSD license.R. Geus    <hamsel@users.sf.net>
D. Orban   <d-orban@users.sf.net>
D. Wheeler <wd15@users.sf.net>
Package: pysparta
SpartaLibrary to help ETL using Pyspark.Sparta is a simple library to help you work on ETL builds using PySpark.Important SourcesApache SparkSmart OpenChispaInstallationInstall the latest version withpip install pyspartaDocumentationSpartaModulesExtractThis is a module with functions for extracting and reading data.Examplefromsparta.extractimportread_with_schemaschema='epidemiological_week LONG, date DATE, order_for_place INT, state STRING, city STRING, city_ibge_code LONG, place_type STRING, last_available_confirmed INT'path='/content/sample_data/covid19-e0534be4ad17411e81305aba2d9194d9.csv'df=read_with_schema(path,schema,{'header':'true'},'csv')TransformationThis is a module with data transformation functionsExamplefromsparta.transformationimportdrop_duplicatescols=['longitude','latitude']df=drop_duplicates(df,'population',cols)LoadThis is a module with load and write functions.Examplefromsparta.loadimportcreate_hive_tablecreate_hive_table(df,"table_name",5,"col1","col2","col3")OthersThis is a module with several functions that can help in ETL work.Examplefromsparta.secretimportget_secret_awsget_secret_aws('Nome_Secret','sa-east-1')Supported PySpark / Python versionsSparta currently supports PySpark 3.0+ and Python 3.7+.
Package: pyspatiotemporalgeom
Source CodePackage source code is hosted at bitbucket:https://bitbucket.org/marmcke/pyspatiotemporalgeom/IntroductionOn open source geometry processing library focusing on regions and moving regions.Algorithms for constructing regions, identifying connected components, creating moving regions, and performing operations on moving regions are included.Some 3D operations are included, such as triangle/triangle intersection and ray/triangle intersection.In general, a regions is represented as a list of segments; a moving region is represented as a list of triangles.This software is open source and distributed under the MIT license.See documentation for details athttp://www.cs.siue.edu/~marmcke/docs/pyspatiotemporalgeom/.The package is hosted at PyPi!InstallingTo install, download the package, unpack, change the working directory to the top level directory of the unpacked directory, and usepython setup.py install.Or useeasy_install pyspatiotemporalgeom.Or use the preferred method:pip install pyspatiotemporalgeom.ContributorsMark McKenneyhttp://www.cs.siue.edu/~marmcke/index.htmlNiharika NyalakondaJarrod McEversMitchell ShiptonJosh Winchester