Saravanan666/pypi-org huggingface dataset:
Package: pyspark
Apache SparkSpark is a unified analytics engine for large-scale data processing. It provides
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing,
and Structured Streaming for stream processing.https://spark.apache.org/Online DocumentationYou can find the latest Spark documentation, including a programming
guide, on theproject web pagePython PackagingThis README file only contains basic information related to pip installed PySpark.
This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility).
Using PySpark requires the Spark JARs, and if you are building this from source please see the builder instructions at"Building Spark".The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to set up your own standalone Spark cluster. You can download the full version of Spark from theApache Spark downloads page.NOTE:If you are using this with a Spark standalone cluster you must ensure that the version (including minor version) matches or you may experience odd errors.Python RequirementsAt its core PySpark depends on Py4J, but some additional sub-packages have their own extra requirements for some features (including numpy, pandas, and pyarrow).
See alsoDependenciesfor production, anddev/requirements.txtfor development.
Package: pyspark3d
The package is under an active development!Latest News[05/2018]GSoC 2018: spark3D has been selected to the Google Summer of Code (GSoC) 2018. Congratulation to@mayurdbwho will work on the project this year![06/2018]Release: version 0.1.0, 0.1.1[07/2018]New location: spark3D is an official project ofAstroLab Software![07/2018]Release: version 0.1.3, 0.1.4, 0.1.5[08/2018]Release: version 0.2.0, 0.2.1 (pyspark3d)[09/2018]Release: version 0.2.2[11/2018]Release: version 0.3.0, 0.3.1 (new DataFrame API)Rationalespark3D should be viewed as an extension of the Apache Spark framework, and more specifically the Spark SQL module, focusing on the manipulation of three*-dimensional data sets.Why would you use spark3D? If you often need to repartition large spatial 3D data sets, or perform spatial queries (neighbour search, window queries, cross-match, clustering, ...), spark3D is for you. It contains optimised classes and methods to do so, and it spares you the implementation time! In addition, a big advantage of all those extensions is to efficiently perform visualisation of large data sets by quickly building a representation of your data set (see morehere).spark3D exposes two API: Scala (spark3D) and Python (pyspark3d). The core developments are done in Scala, and interfaced with Python using the greatpy4jpackage. This means pyspark3d might not contain all the features present in spark3D.
In addition, due to difference between Scala and Python, there might be subtle differences in the two APIs.While we try to stick to the latest Apache Spark developments, spark3D started with the RDD API and slowly migrated to use the DataFrame API. This process left a huge imprint on the code structure, and low-level layers in spark3D often still use RDD to manipulate the data. Do not be surprised if things are moving, the package is under an active development but we try to keep the user interface as stable as possible!Last but not least: spark3D is by no means complete, and you are welcome to suggest changes, report bugs or inconsistent implementations, and contribute directly to the package!Cheers,
JulienWhy 3? Because there are already plenty of very good packages dealing with 2D data sets (e.g.geospark,geomesa,magellan,GeoTrellis, and others), but that was not suitable for many applications such as in astronomy!Installation and tutorialsScalaYou can link spark3D to your project (either spark-shell or spark-submit) by specifying the coordinates:spark-submit --packages "com.github.astrolabsoftware:spark3d_2.11:0.3.0"PythonJust runpipinstallpyspark3dNote that we release the assembly JAR with it.More informationSee ourwebsite!ContributorsJulien Peloton (peloton at lal.in2p3.fr)Christian Arnault (arnault at lal.in2p3.fr)Mayur Bhosale (mayurdb31 at gmail.com) -- GSoC 2018.Contributing to spark3D: seeCONTRIBUTING.Support
Package: py-sparkblocks
About sparkblockssparkblocksis a Python module that provides a single function,spark()which takes a list of numbers (ints or floats) and outputs a string containing a unicode character (9601-9608) sparklines bargraph representation of the numbers.It is based on thespark toolby Zach Holman, and the originalTwitter sparkline generatorby Alex Kerin.InstallationThe library can be installed from PyPi usingpip:pip install py-sparkblocksOr cloned fromGithubusinggit:git clone git://github.com/1stvamp/py-sparkblocks.git
cd py-sparkblocks
python setup.py installUsage>>> from sparkblocks import spark
>>> print(spark([1, 2, 0.2, 3.1, 2]))
▂▅▁▇▅
Package: pyspark-bucketmap
pyspark-bucketmappyspark-bucketmapis a tiny module forpysparkwhich allows you to bucketize DataFrame rows and map their values easily.Installpipinstallpyspark-bucketmapUsagefrompyspark.sqlimportRowpeople=spark.createDataFrame([Row(age=12,name="Damian"),Row(age=15,name="Jake"),Row(age=18,name="Dominic"),Row(age=20,name="John"),Row(age=27,name="Jerry"),Row(age=101,name="Jerry's Grandpa"),])peopleNow, what we would like to do, is map each person's age to an age category.age rangelife phase0 to 12Child12 to 18Teenager18 to 25Young adulthood25 to 70Adult70 and beyondElderlyWe can usepyspark-bucketmapfor this. First, define the splits and mappings:fromtypingimportListsplits:List[float]=[-float("inf"),0,12,18,25,70,float("inf")]mapping:Dict[int,Column]={0:lit("Not yet born"),1:lit("Child"),2:lit("Teenager"),3:lit("Young adulthood"),4:lit("Adult"),5:lit("Elderly"),}Then, applyBucketMap.transform(df):frompyspark_bucketmapimportBucketMapfromtypingimportList,Dictbucket_mapper=BucketMap(splits=splits,mapping=mapping,inputCol="age",outputCol="phase")phases_actual:DataFrame=bucket_mapper.transform(people).select("name","phase")phases_actual.show()namephaseDamianTeenagerJakeTeenagerDominicYoung adulthoodJohnYoung adulthoodJerryAdultJerry's GrandpaElderlySuccess!✨APIModulepyspark_bucketmap:from pyspark.ml.feature import Bucketizer
from pyspark.sql import DataFrame
from pyspark.sql.column import Column

class BucketMap(Bucketizer):
    mapping: Dict[int, Column]

    def __init__(self, mapping: Dict[int, Column], *args, **kwargs):
        ...

    def transform(self, dataset: DataFrame, params: Optional[Any] = None) -> DataFrame:
        ...ContributingUnder the hood, uses a combination of pyspark'sBucketizerandpyspark.sql.functions.create_map. The code is 42 lines and exists 1 in file:pyspark_bucketmap.py. To contribute, follow your preferred setup option below.Option A: using a Devcontainer (VSCode only)If you happen to use VSCode as your editor, you can openfsevalin aDevcontainer. Devcontainers allow you to developinsidea Docker container - which means all dependencies and packages are automatically set up for you. First, make sure you have theRemote Development extensioninstalled.Then, you can do two things.Click the following button:Or, clone and open up the repo in VSCode:gitclonehttps://github.com/dunnkers/pyspark-bucketmap.git
codepyspark-bucketmap(for this to work, make sure you activated VSCode'scodeCLI)Then, you should see the following notification:Now you should have a fully working dev environment working 🙌🏻. You can run tests, debug code, etcetera. All dependencies are automatically installed for you.🙌🏻Option B: installing the dependencies manuallyClone the repo and install the deps:git clone https://github.com/dunnkers/pyspark-bucketmap.git
cd pyspark-bucketmap
pip install -r .devcontainer/requirements.txt
pip install -r .devcontainer/requirements-dev.txt
pip install .Make sure you also have the following installed:Python 3.9OpenJDK version 11Now, you should be able to run tests 🧪:pytest .🙌🏻AboutCreated byJeroen Overschie© 2022.
Package: pysparkcli
PySpark CLIThis will implement a PySpark Project boiler plate code based on user input.Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.PySpark is the Python API for Spark.Installation Steps:git clone https://github.com/qburst/PySparkCLI.git

cd PySparkCLI

pip3 install -e . --userCreate a PySpark Projectpysparkcli create [PROJECT_NAME] --master [MASTER_URL] --cores [NUMBER]

master - The URL of the cluster it connects to. You can also use -m instead of --master.
cores - You can also use -c instead of --cores.Run a PySpark Projectpysparkcli run [PROJECT_NAME]PySpark Project Test casesRunning byProject namepysparkcli test [PROJECT_NAME]Running individual test case with filename:test_etl_job.pypysparkcli test [PROJECT_NAME] -t [etl_job]FAQCommon issues while installing pysparkcli:* pysparkcli: command not found
    Make sure you add user’s local bin to PATH variable.
    Add the following code in .bashrc file

    # set PATH so it includes user's private bin if it exists
    if [ -d "$HOME/.local/bin" ] ; then
        PATH="$HOME/.local/bin:$PATH"
    fi


* JAVA_HOME is not set
    Make sure JAVA_HOME is pointing to your JDK and PYSPARK_PYTHON variable is created.
    You can add them manually by in .bashrc file:

    Example:

        export PYSPARK_PYTHON=python3
        export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

    Save the file and run the following to update environment.

        source ~/.bashrcProject StructureThe basic project structure is as follows:sample
├──__init__.py
├──src
│├──app.py
│├──configs
││├──etl_config.json
││└──__init__.py
│├──__init__.py
│├──jobs││├──etl_job.py
││└──__init__.py
│└──settings
│├──default.py
│├──__init__.py
│├──local.py
│└──production.py
└──tests├──__init__.py├──test_data│├──employees││└──part-00000-9abf32a3-db43-42e1-9639-363ef11c0d1c-c000.snappy.parquet│└──employees_report│└──part-00000-4a609ba3-0404-48bb-bb22-2fec3e2f1e68-c000.snappy.parquet└──test_etl_job.py8directories,15filesContribution GuidelinesCheck outherefor our contribution guidelines.Sponsors
Package: pyspark-cli
PySpark CLIThis will implement a PySpark Project boiler plate code based on user input.Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.PySpark is the Python API for Spark.Installation Steps:git clone https://github.com/qburst/PySparkCLI.git

cd PySparkCLI

pip3 install -e . --userCreate a PySpark Projectpysparkcli create [PROJECT_NAME] --master [MASTER_URL] --cores [NUMBER]

master - The URL of the cluster it connects to. You can also use -m instead of --master.
cores - You can also use -c instead of --cores.Run a PySpark Projectpysparkcli run [PROJECT_NAME]Initiate Stream for Projectpysparkcli stream [PROJECT_NAME] [STREAM_FILE_NAME]PySpark Project Test casesRunning byProject namepysparkcli test [PROJECT_NAME]Running individual test case with filename:test_etl_job.pypysparkcli test [PROJECT_NAME] -t [etl_job]FAQCommon issues while installing pysparkcli:* pysparkcli: command not found
    Make sure you add user’s local bin to PATH variable.
    Add the following code in .bashrc file

    # set PATH so it includes user's private bin if it exists
    if [ -d "$HOME/.local/bin" ] ; then
        PATH="$HOME/.local/bin:$PATH"
    fi


* JAVA_HOME is not set
    Make sure JAVA_HOME is pointing to your JDK and PYSPARK_PYTHON variable is created.
    You can add them manually by in .bashrc file:
    
    Example:

        export PYSPARK_PYTHON=python3
        export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

    Save the file and run the following to update environment.

        source ~/.bashrcProject StructureThe basic project structure is as follows:sample
├──__init__.py
├──src
│├──app.py
│├──configs
││├──etl_config.json
││└──__init__.py
│├──__init__.py
│├──jobs││├──etl_job.py
││└──__init__.py
│└──settings
│├──default.py
│├──__init__.py
│├──local.py
│└──production.py
└──tests├──__init__.py├──test_data│├──employees││└──part-00000-9abf32a3-db43-42e1-9639-363ef11c0d1c-c000.snappy.parquet│└──employees_report│└──part-00000-4a609ba3-0404-48bb-bb22-2fec3e2f1e68-c000.snappy.parquet└──test_etl_job.py8directories,15filesPySparkCLI DemoContribution GuidelinesCheck outherefor our contribution guidelines.Sponsors
Package: pyspark-cloud
Tekumara build of Apache PySpark with Hadoop 3.xA build of Apache PySpark that uses the hadoop-cloud maven profile to bundlehadoop-aws 3.xwhich contains S3A.InstallSeeReleasesUsageTo use pyspark with temporary STS credentials:pyspark --driver-java-options "-Dspark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider"To modify an existing spark session to use S3A for S3 urls, for examplesparkin the pyspark shell:spark.sparkContext._jsc.hadoopConfiguration().set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")Seetest_s3a.pyfor an example of using the staging committers.RationaleThepyspark distribution on pypiships with hadoop 2.7 and no cloud jars (ie: hadoop-aws).
So common practice is to use hadoop-aws 2.7.3 as follows:pyspark --packages "org.apache.hadoop:hadoop-aws:2.7.3" --driver-java-options "-Dspark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem"However, later versions of hadoop-aws cannot be used this way without errors.This projectbuilds a pyspark distributionfrom source with Hadoop 3.x.Later versions of hadoop-aws contain the following new features:2.8 release linecontains S3A improvements to support any AWSCredentialsProvider2.9 release linecontainsS3Guardwhich provides consistency and metadata caching for S3A via a backing DynamoDB metadata store.3.1 release lineincorporates HADOOP-13786 which contains optimised job committers including the Netflix staging committers (Directory and Partitioned) and the Magic committers. Seecommittersandcommitter architecture.3.2 release lineanenhanced S3A connector and S3Guard, including better resilience to throttled AWS S3 and DynamoDB IO.To take advantage of the 3.x release line committers in Spark you also need the binding classes introduced into Spark 3.0.0 bySPARK-23977. For Spark 2.4, theHortonWorks backportis used from theHortonworks repo.
Package: pyspark-config
Pyspark-configPyspark-Config is a Python module for data processing in Pyspark by means of a configuration file, granting access to build distributed data piplines with configurable inputs, transformations and outputs.Getting StartedThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.InstallationTo install the current release(Ubuntu and Windows):$ pip install pyspark_configDependenciesPython (>= 3.6)Pyspark (>= 2.4.5)PyYaml (>= 5.3.1)Dataclasses (>= 0.0.0)ExampleGiven the yaml configuration file '../example.yaml':input:sources:-type:'Parquet'label:'parquet'parquet_path:'../table.parquet'transformations:-type:"Select"cols:['A','B']-type:"Concatenate"cols:['A','B']name:'Concatenation_AB'delimiter:"-"output:-type:'Parquet'name:"example"path:"../outputs"With the input source saved in '../table.parquet', the following code can then be applied:frompyspark_configimportConfigfrompyspark_config.transformations.transformationsimport*frompyspark_config.outputimport*frompyspark_config.inputimport*config_path="../example.yaml"configuration=Config()configuration.load(config_path)configuration.apply()The output will then be saved in '../outputs/example.parquet'.ChangelogSee the changelog for a history of notable changes to pyspark-config.LicenseThis project is distributed under the 3-Clause BSD license. - see theLICENSE.mdfile for details.
Package: pyspark-connectby
pyspark-connectbySpark currently does not support hierarchy queryconnectByas of version 3.5.0. And there is aPRopened to support recursive CTE query. But that is still not available yet.This is an attempt to addconnectBymethod toDataFrameConceptHierarchy query is one of the important feature that many relational databases, such as Oracle, DB2, My SQL,
Snowflake,Redshift, etc.,
would support directly or alternatively by using recursive CTE.Example in Redshift:selectid,name,manager_id,levelfromemployeestartwithemp_id=1connectbyprioremp_id=manager_id;With this library, you can useconnectBy()onDateframe:frompyspark_connectby.queryimportconnectByfrompyspark.sqlimportSparkSessionschema='emp_id string, manager_id string, name string'data=[[1,None,'Carlos'],[11,1,'John'],[111,11,'Jorge'],[112,11,'Kwaku'],[113,11,'Liu'],[2,None,'Mat']]df=spark.createDataFrame(data,schema)df2=df.connectBy(prior='emp_id',to='manager_id',start_with='1')df2.show()With result:+------+-----+----------+------+
|emp_id|level|manager_id|  name|
+------+-----+----------+------+
|     1|    1|      null|Carlos|
|    11|    2|         1|  John|
|   111|    3|        11| Jorge|
|   112|    3|        11| Kwaku|
|   113|    3|        11|   Liu|
+------+-----+----------+------+InstallationPython Version >= 3.7$ pip install --upgrade pyspark-connectbyUsagefrompyspark_connectby.queryimportconnectBydf=...df.connectBy(prior='emp_id',to='manager_id',start_with='1')# start_with `emp_id` as 1df.transform(connectBy,prior='emp_id',to='manager_id',start_with='1')# or by using df.transform() methoddf.connectBy(prior='emp_id',to='manager_id')# without start_with, it will go through each nodedf.connectBy(prior='emp_id',to='manager_id',level_col='the_level')# level column name other than `level`df.connectBy(prior='emp_id',to='manager_id',start_with=['1','2'])# start_with a list of top nodes ids.
Package: pySpark-connector-kairosdb
pySpark-connector-kairosdb provides you an easy way to get data from KairosDB and make it available on Spark as a DataFrame. It’s simple as that:#!/usr/bin/env python

# sconnk means "py(S)park-(CONN)ector-(K)airosdb"
from sconnk import Connection, Dataframe

query_data = {
                    "start_relative": { "value": "5", "unit": "years" },
                    "metrics": [{ "name": "test_", "limit": 5 },
                                            { "name": "DP_058424", "limit": 10 },
                                            { "name": "teste_gzip", "limit": 5 },
                                            { "name": "DP_063321", "limit": 10 }]
    }

    # Creating a connection with KairosDB database (in the given address).
    conn = Connection("http://localhost:8080/")

    # Performing our query on KairosDB.
    json_data = conn.query(query_data)

    # Creating a new Dataframe object passing the JSON returned by KairosDB API.
    df = Dataframe(json_data).df

    # Print the dataframe.
    df.show()Remember this’s a ALPHA module without good documentation, examples and even well implemented features. We’ve a long highway to cross.FutureThis module is in development and we’ve the following plannings for the future of this module:Write good documentationWrite tests - a lot of them.Add support to RDDDon’t write to JSON file in order to parse it in Spark.
Package: pyspark-connectors
PySpark Conectors - by: Eleflow BigDataThis library provides many connections and integrations with another data sources.IndexPySpark Conectors - by: Eleflow BigDataIndexInstallingDevelopment enviromentPackaging project in a .whl libLibrary development statusConnectorsHelpers and UtilsVersion historyInstallingpipinstallpyspark-connectorsDevelopment enviromentFor develop you must guarantee that you have the Python (3.8 or higher) and Spark (3.1.2 or higher) installed, if you have ready the minimum environment for development in Python language, proceed with these steps:# Clonning the project$gitclonegit@github.com:eleflow/pyspark-connectors.git# Inside of the project root folder$python-mvenv.env# If Windows$.\.env\Script\Activate.ps1# If linux dist$.\.env\Scripts\activate# Installing requirements libraries(.env)$pipinstall-r.\requirements.txtPackaging project in a .whl lib# Installing wheel package(.env)$pipinstallwheel# Installing wheel contents(.env)$pipinstallcheck-wheel-contents# Build and packaging project to .whl(.env)$pythonsetup.pybdist_wheelLibrary development statusConnectorsGoogle SheetsRest APISQL DatabaseCosmosDBElasticsearchPipeDriveActiveCampaignReclameAquiJiraHelpers and UtilsAWS Secrets ManagerVersion historyVersionDateChangesNotesApproved by0.0.1a22022-05-08Initial development releaseN/A@caiodearaujo0.1.02022-06-01Initial releaseN/A@caiodearaujo0.2.02022-07-28New release with connectors stableN/A@caiodearaujo
Package: pyspark-custom-types-patch
COPY FILES FROMApache SparkWorkaround to add new types to dataframeAll code left as it is were in pyspark3.0.1 version except few lines in types.py and dataframe.py from sql folderReplace type.py and dataframe.py in pyspark/sql folder with ones from this packagesuits for pyspark3.0.x versionSpark is a unified analytics engine for large-scale data processing. It provides
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
MLlib for machine learning, GraphX for graph processing,
and Structured Streaming for stream processing.https://spark.apache.org/Online DocumentationYou can find the latest Spark documentation, including a programming
guide, on theproject web pagePython PackagingThis README file only contains basic information related to pip installed PySpark.
This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility).
Using PySpark requires the Spark JARs, and if you are building this from source please see the builder instructions at"Building Spark".The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to set up your own standalone Spark cluster. You can download the full version of Spark from theApache Spark downloads page.NOTE:If you are using this with a Spark standalone cluster you must ensure that the version (including minor version) matches or you may experience odd errors.Python RequirementsAt its core PySpark depends on Py4J, but some additional sub-packages have their own extra requirements for some features (including numpy, pandas, and pyarrow).