Saravanan666/pypi-org huggingface dataset:
Package: pyspark-data-sources
pyspark-data-sourcesThis repository showcases custom Spark data sources built using the newPython Data Source APIfor the upcoming Apache Spark 4.0 release.
For an in-depth understanding of the API, please refer to theAPI source code.Installationpip install pyspark-data-sourcesUsageNote: Currently the following code only works with Apache Sparkmasterbranch.frompyspark_datasources.githubimportGithubDataSource# Register the data sourcespark.dataSource.register(GithubDataSource)spark.read.format("github").load("apache/spark").show()ContributingWe welcome and appreciate any contributions to enhance and expand the custom data sources. If you're interested in contributing:Add New Data Sources: Want to add a new data source using the Python Data Source API? Submit a pull request or open an issue.Suggest Enhancements: If you have ideas to improve a data source or the API, we'd love to hear them!Report Bugs: Found something that doesn't work as expected? Let us know by opening an issue.Need help or have questions?Don't hesitate to open a new issue, and we'll do our best to assist you.
Package: pyspark-dbscan
pyspark_dbscanAn Implementation of DBSCAN on PySparkimportdbscanfromsklearn.datasetsimportmake_blobsfrompyspark.sqlimporttypesasT,SparkSessionfromscipy.spatialimportdistancespark=SparkSession\.builder\.appName("DBSCAN")\.config("spark.jars.packages","graphframes:graphframes:0.7.0-spark2.3-s_2.11")\.config('spark.driver.host','127.0.0.1')\.getOrCreate()X,labels_true=make_blobs(n_samples=750,centers=centers,cluster_std=0.4,random_state=5)data=[(i,[float(item)foriteminX[i]])foriinrange(X.shape[0])]schema=T.StructType([T.StructField("id",T.IntegerType(),False),T.StructField("value",T.ArrayType(T.FloatType()),False)])#please repartition appropriatelydf=spark.createDataFrame(data,schema=schema).repartition(10)df_clusters=dbscan.process(spark,df,.2,10,distance.euclidean,2,"checkpoint")
Package: pyspark_db_utils
# pyspark_db_utilsIt helps you with your DB deals in Spark## Documentationhttp://pyspark-db-utils.readthedocs.io/en/latest/## Example of usingYou need jdbc drivers for using this lib!Just get drivers fromhttps://jdbc.postgresql.org/download.htmlhttps://github.com/yandex/clickhouse-jdbcand put it in jars/ directory in your project### Example settings:```settings = {"PG_PROPERTIES": {"user": "user","password": "pass","driver": "org.postgresql.Driver"},"PG_DRIVER_PATH": "jars/postgresql-42.1.4.jar","PG_URL": "jdbc:postgresql://db.olabs.com/dbname",}```### Example of codesee example.py### Example of run```vsmelov@vsmelov:~/PycharmProjects/pyspark_db_utils$ mkdir jarsvsmelov@vsmelov:~/PycharmProjects/pyspark_db_utils$ cp /var/bigdata/spark-2.2.0-bin-hadoop2.7/jars/postgresql-42.1.4.jar ./jars/vsmelov@vsmelov:~/PycharmProjects/pyspark_db_utils$ python3 pyspark_db_utils/example.pyhost: ***SECRET***db: ***SECRET***user: ***SECRET***password: ***SECRET***Using Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).18/03/05 11:43:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable18/03/05 11:43:29 WARN Utils: Your hostname, vsmelov resolves to a loopback address: 127.0.1.1; using 192.168.43.26 instead (on interface wlp2s0)18/03/05 11:43:29 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another addressTRY: create dfOK: create df+---+-----------+| id|    mono_id|+---+-----------+|  1|          0||  2|          1||  3|          2||  4|          3||  5| 8589934592||  6| 8589934593||  7| 8589934594||  8| 8589934595||  9| 8589934596|| 10|17179869184|| 11|17179869185|| 12|17179869186|| 13|17179869187|| 14|17179869188|| 15|25769803776|| 16|25769803777|| 17|25769803778|| 18|25769803779|| 19|25769803780|+---+-----------+TRY: write_to_pgOK: write_to_pgTRY: read_from_pgOK: read_from_pg+---+-----------+| id|    mono_id|+---+-----------+| 10|17179869184|| 11|17179869185|| 12|17179869186|| 13|17179869187|| 14|17179869188||  1|          0||  2|          1||  3|          2||  4|          3||  5| 8589934592||  6| 8589934593||  7| 8589934594||  8| 8589934595||  9| 8589934596|| 15|25769803776|| 16|25769803777|| 17|25769803778|| 18|25769803779|| 19|25769803780||  1|          0|+---+-----------+only showing top 20 rows```
Package: pyspark-delta-scd2
Demo PySpark Delta Table SCD2 implementationThis project utilizesfaker-pysparkto generate random schema and dataframes to mimic data table snapshots.Using these snapshots to process and apply SCD2 pattern into delta table as the destination.Source of Inspiration for SCD2 pattern:https://aws-blogs-artifacts-public.s3.amazonaws.com/artifacts/BDB-2547/glue/scd-deltalake-employee-etl-job.pyInstallationInstall with pip:pipinstallpyspark-delta-scd2delta-sparkfaker-pysparkPlease note, this package do not enforce version of delta-spark, PySpark and faker-pyspark.When you want to use this example in AWS Glue environment, enforced versions conflict with the target environment.Generate incremental updates to dataframe and apply scd2>>>frompyspark_delta_scd2importget_spark,PySparkDeltaScd2>>>spark=get_spark()>>>demo=PySparkDeltaScd2(spark=spark)>>># initial load>>>df1=demo.process()>>># incremental update>>>df2=demo.process()>>># df2 should have some deletes, updates and inserts
Package: pyspark-delta-utility
pyspark_delta_utilityA useful python package that consists of pyspark and delta utilitiesInstallation$pipinstallpyspark_delta_utilityUsageTODOContributingInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.Licensepyspark_delta_utilitywas created by Kyle Ahn. It is licensed under the terms of the MIT license.Creditspyspark_delta_utilitywas created withcookiecutterand thepy-pkgs-cookiecuttertemplate.
Package: pyspark-ds-toolbox
Pyspark DS ToolboxThe objective of the package is to provide a set of tools that helps the daily work of data science with spark. The documentation can be foundhereand notebooks with usage exampleshere.Feel free to contribute :)InstallationDirectly from PyPi:pip install pyspark-ds-toolboxor from github, note that installing from github will install the latest development version:pip install git+https://github.com/viniciusmsousa/pyspark-ds-toolbox.gitOrganizationThe package organized in a structure based on the nature of the task, such as data wrangling, model/prediction evaluation, and so on.pyspark_ds_toolbox         # Main Package
├─ causal_inference           # Sub-package dedicated to Causal Inferece
│  ├─ diff_in_diff.py   
│  └─ ps_matching.py    
├─ ml                         # Sub-package dedicated to ML
│  ├─ data_prep                  # Sub-package to ML data preparation tools
│  │  ├─ class_weights.py     
│  │  └─ features_vector.py 
│  ├─ classification             # Sub-package decidated to classification tasks
│  │  ├─ eval.py
│  │  └─ baseline_classifiers.py 
│  ├─ feature_importance         # Sub-package with feature importance tools
│  │  ├─ native_spark.py
│  │  └─ shap_values.py 
│  └─ feature_selection         # Sub-package with feature selection tools
│     └─ information_value.py    
├─ wrangling                  # Sub-package decidated to data wrangling tasks
│  ├─ reshape.py               
│  └─ data_quality.py         
└─ stats                      # Sub-package dedicated to basic statistic functionalities
   └─ association.py
Package: pysparketl
Pyspark ETLThis project aims at solving common problems that face data engineers.One problem is to handle deeply nested json data and render data in a clean tabular format.This kind of semistructured data may contain a combination of different data types that need to be handled differently to flatten the data properly.This package does just that!In this initial version of the package, there is one module namedpysparketlwhich has one main function:flattenDFand two utility functions:_getArrayCols,_explodeArrayCols.UsageflattenDF(df) where df is a pyspark dataframe that has nested data in its columns.
The returned dataframe will be a completely flat/tabular structure.ExampleInstall the Packagepipinstallpysparketl
<!--OR-->
pip3installpysparketlImport and Use# Import the libraryfrompysparketl.dataframesimportflattenDF# Read your input dataframe that contains Json datarawJsonDF=spark.read.json(pathToFile)# Pass your nested json dataframe as input to the flattenDF functionflattenedDF=flattenDF(rawJsonDF)# flattenedDF will be a flat dataframe instead as a result.
Package: pyspark-event-correlation
pyspark-event-correlationEvent Correlation and Changing Detection AlgorithmTheoryCorrelating events in complex and dynamic IoT environments is a challenging task not only because of the amount of available data that needs to be processed but also due to the call for time efficient data processing. We propose the adoption of a univariate change detection algorithm for real-time event detection and we implement a stepwise event correlation scheme based on a first-order Markov model.RequirementsPython 3.6 to 3.10 supported.pyspark 3.2.1 supported.InstallationInstall with pip:python -m pip install pyspark-event-correlationExample ProjectSee the example project in the example/ directory of the GitHub repository.
Package: pysparkextra
pyspark extra utilitiesSparkMetricsTrack metrics (like number of rows/number of files written) when writing a DataFrame.importpyspark.sqlfrompysparkextra.metricsimportSparkMetricsspark_session:pyspark.sql.SparkSessiondf:pyspark.sql.DataFrame=spark_session.createDataFrame([[1,2],[-3,4],],schema=("foo","bar"))withSparkMetrics(spark_session)asmetrics:df.write.parquet("/tmp/target",mode='overwrite')print(metrics['numOutputRows'])# 2withSparkMetrics(spark_session)asmetrics:df.union(df).write.parquet("/tmp/target",mode='overwrite')print(metrics['numOutputRows'])# 4print(metrics)# {'numFiles': 5, 'numOutputBytes': 3175, 'numOutputRows': 4, 'numParts': 0}union arbitrary number of dataframes with arbitrary number of columnsfrompyspark.sqlimportDataFrame,SparkSessionfrompysparkextra.funcsimportunionspark_session:SparkSessiondf1:DataFrame=spark_session.createDataFrame([[1,2],[3,4],],schema=("foo","bar"))df2:DataFrame=spark_session.createDataFrame([[10,20,30],[40,50,60],],schema=("bar","qux","foo"))df3:DataFrame=spark_session.createDataFrame([[100,200],[300,400],],schema=("foo","bar"))df:DataFrame=union(df1,df2,df3)df.show()# +---+---+----+# |foo|bar| qux|# +---+---+----+# |  1|  2|null|# |  3|  4|null|# | 30| 10|  20|# | 60| 40|  50|# |100|200|null|# |300|400|null|# +---+---+----+and moreCheck out the tests, which also act as examples.
Package: pyspark-flame
pyspark-flameA low-overhead profiler for Spark on PythonPyspark-flame hooks into Pyspark's existing profiling capabilities to provide a
low-overhead stack-sampling profiler, that outputs performance data in a
format compatible withBrendan Gregg's FlameGraph Visualizer.Because pyspark-flame hooks into Pyspark's profiling capabilities, it can profile
the entire execution of an RDD, across the whole of the cluster, and provides
RDD-level visibility of performance.Unlike the cProfile-based profiler included with Pyspark, pyspark-flame uses
stack sampling. It takes stack traces at regular (configurable) intervals,
which allows its overhead to be low and tunable, and doesn't skew results,
making it suitable for use in performance test environments at high volumes.Installationpipinstallpyspark-flameUsagefrompyspark_flameimportFlameProfilerfrompysparkimportSparkConf,SparkContextconf=SparkConf().set("spark.python.profile","true")conf=conf.set("spark.python.profile.dump",".")# Optional - if not, dumps to stdout at exitsc=SparkContext('local','test',conf=conf,profiler_cls=FlameProfiler,environment={'pyspark_flame.interval':0.25}# Optional - default is 0.2 seconds)# Do stuff with Spark context...sc.show_profiles()# Or maybesc.dump_profiles('.')For convenience, flamegraph.pl is vendored in, so you can produce a flame graph
with:flamegraph.plrdd-1.flame>rdd-1.svg
Package: pyspark-helpers
PySpark HelpersA collection of tools to help when developing PySpark applicationsInstallationWith pippip install pyspark_helpersWith poetrypoetry add pyspark_helpersUsageAuto Generate PySpark Schemas from JSON examplesThrough cli:python-mpyspark_helpers.schema# OR with scriptpsh-schema-from-json--path./tests/data/schema/array.json--output./results/array_schema.jsonOr programaticallyfrompyspark_helpers.schemaimportschema_from_json,bulk_schema_from_jsomfrompathlibimportPathdata_dir="data/json"## One fileschema=schema_from_json(f"{data_dir}/file.json")print(schema)## A whole directoryfiles=[Path(f)forfinPath.glob(f"{data_dir}/*.json")]schemas=bulk_schema_from_jsom(files)for_file,schemainzip(files,schemas):print(_file.name,schema)Guidelies for ContributingUseConventional Commitmessages.To help with this, I encourage you to use commitizen when making your commits. The process for this is straight forward:# Checkout a new branchgitcheckout-b<my-new-branch># Make changes to the code....# Add your changesgitadd<changed-file-1><changed-file-2>...# Run commitizen commit and follow promptscommitizencommit# `cz c` in short# Push branchgitpush# Open a Pull Request## This is done in Github
Package: pysparkify
IntroductionThis Spark package is designed to process data from various sources, perform transformations, and write the results to different sinks. It follows the pipeline design pattern to provide a flexible and modular approach to data processing.InstallThe package is published for use on pypi. Install directly from pypipip install pysparkifyDesignThe package is structured as follows:Source, Sink and Transformer AbstractionThe package defines abstract classesSource,SinkandTransformerto represent data sources, sinks and transformers. It also provides concrete classes, includingCsvSource,CsvSinkandSQLTransformer, which inherit from the abstract classes. This design allows you to add new source and sink types with ease.Configuration viarecipe.ymlThe package reads its configuration from arecipe.ymlfile. This YAML file specifies the source, sink, and transformation configurations. It allows you to define different data sources, sinks, and transformation queries.Transformation QueriesTransformations are performed bySQLTransformerusing Spark SQL queries defined in the configuration. These queries are executed on the data from the source before writing it to the sink. New transformers can be implemented by extendingTransformerabstract class that can take spark dataframes from sources to process and send dataframes to sinks to save.Pipeline ExecutionThe package reads data from the specified source, performs transformations based on the configured SQL queries, and then writes the results to the specified sink. You can configure multiple sources and sinks within the same package.SetupThe project is built using python-3.12.0, spark-3.5.0 (and other dependencies in requirements.txt).Deployment... Environment specific documentation
Package: pyspark-iomete
Pyspark IOMETE LibraryThis library is providing a set of utility functions to speed up the development of pyspark applications.Installationpipinstallpyspark-iometeUtility functionsget_spark_loggerThis function is returning a spark logger instance.As you may know, spark is using log4j as a logging framework. This function is returning a spark logger instance that is using the log4j logger. Standard python logging is not working with pyspark. The following function get the spark logger instance and returns it.Usage:frompyspark_iomete.utilsimportget_spark_loggerfrompyspark.sqlimportSparkSessionspark=SparkSession.builder.getOrCreate()# spark session and name will be used to create the logger# both are optionallogger=get_spark_logger(spark=spark,name="my_custom_logger")# spark session will be retrieved using SparkSession.getActiveSession() and name will be set to the current file namelogger=get_spark_logger()Test utility functionstable_name_with_random_suffixThis function is returning a table name with a random suffix. This is useful for testing purposes.Usage:frompyspark_iomete.test_utilsimporttable_name_with_random_suffixtable_name=table_name_with_random_suffix("my_table")
Package: pyspark-json-model
Easily Parse your JSON & convert to relational data model format. This is currenlty build to support on Databricks/PysparkDescriptionThis package if for converting any json formatted data to relational data model in dataframe format for databricksjsontools: tools to parse json & converts to relational data model formatInstallationNormal installationpipinstallpyspark_json_modelDevelopment installationgitclonehttps://github.com/rohitpawar95/pyspark_json_model.gitcdpyspark_json_model
pipinstall--editable.Usagefrom pyspark_json_model import *

json2model_obj = jsontools.JSONParserPyspark('filepath_of_json', 
			<identifier_for_json_document:str>, 
		 	<is_looking_for_normalized_dataframe:bool>)

normalized_json_model = json2model_obj.process(sc, spark, ['drop_columns',...])

display(normalized_json_model[key])

Note: 
	is_looking_for_normalized_dataframe --> if True : The returned normalized_json_model 
	will hold extra key for root_node+'_normalized' which will have normalized json 
	format (This will be same as json_normalize function of pandas)

	is_looking_for_normalized_dataframe --> if False : The returned normalized_json_model 
	will just contain multiple keys which will hold independent tables of data model 
		
	normalized_json_model : This will be dictionary of data model key will be table name 
	and value will be dataframe corresponding to this key
	
	Pass sparkContext (sc) & sparkSession (spark) to process JSON along with 
	['drop_columns'] : The parameter to process() function is optional if need to remove 
	any columns while making this dataframe specify column list hereChange log[0.0.3] - 2021-10-17AddedChangedUpdated process function to handle UTF-8 BOM json formatRemoved[0.0.2] - 2021-10-17AddedChangedUpdated process function to handle Array & Dict data typesRemoved[0.0.1] - 2021-10-14AddedJSONParserPyspark.array_identification()JSONParserPyspark.check_all_done()JSONParserPyspark.get_nested_field()JSONParserPyspark.explode_struct_field()JSONParserPyspark.explode_array_df()JSONParserPyspark.add_random_id()JSONParserPyspark.main()JSONParserPyspark.drop_columns()JSONParserPyspark.drop_all_duplicates()JSONParserPyspark.remove_df()JSONParserPyspark.check_relationship()JSONParserPyspark.join_multiple_df()JSONParserPyspark.normalized_json()JSONParserPyspark.process()ChangedRemoved
Package: pyspark-kernel
# pyspark_kernelA PySpark Jupyter kernel that utilizes [metakernel](https://github.com/Calysto/metakernel) to create an easy to initialize pyspark kernel for Python. This package resembles the [spylon-kernel](https://github.com/vericast/spylon-kernel) that utilizes metakernel as well.## PrerequisitesApache SparkJupyter NotebookPython 3.5+## Install PackageYou can install the pyspark_kernel package usingpip.`bash pip install pyspark_kernel `## Install Kernel to use in JupyterTo use pyspark_kernel as PySpark kernel for Jupyter Notebook run the following command:`bash python-mpyspark_kernel install `Once Jupyter launches and you should seePySparkas an option in theNewdropdown menu.## Packaging
To package to deploy simply run the following command from the top level of the package.`bash python setup.py sdist `
Package: pysparklib
Copyright (c) 2019 Deyou Lee <deyoulee@126.com>Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.Description: UNKNOWN
Keywords: apache,spark,sparkMl,pmml
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Chinese (Simplified)
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.7
Package: pysparklines
pysparklinePython 3 clone ofZach Holman's BASH sparkline projectTakes series data via stdin, command line, or API and prints a sparkline representation.Usage:$sparkline 4 3 2 1█▆▃▁$echo "1.0 1.0 2.0 3.0 5.0 8.0 13.0" | sparkline▁▁▂▂▃▅█$seq 20 | sort -R | sparkline -r2▃▁ █▂    ▂▆▅  ▄▇▆   
██▃██▅▇▄▁███▃▇███▂█▆$python3 -c "import sparkline; print(sparkline.sparkify([1.0, 2.0, 3.0, 4.0]))"▁▃▆█$python3 -c "import math, sparkline; print(sparkline.sparkify([math.cos(n/10.0) for n in range(-50, 50, 2)], rows=4))"▁▃▅▇███▇▅▃▁                   
▅▂                ▃▆███████████▆▃                ▂
██▇▄▁          ▂▅█████████████████▅▂          ▁▄▇█
█████▆▄▃▁▁▁▂▃▅▇█████████████████████▇▅▃▂▁▁▁▃▄▆████