Saravanan666/pypi-org huggingface dataset:
Package: dagsim
DagSimDagSim is a Python-based framework and specification language for simulating data based on a Directed Acyclic Graph (
DAG)
structure, without any constraints on variable types or functional relations. A succinct YAML format for
defining the structure of the simulation model promotes transparency, while separate user-provided functions for
generating each variable based on its parents ensure the modularization of the simulation code.InstallationDagSim can be easily installed using pip.Installing DagSim usingpipTo install the DagSim package usingpip, run:pipinstalldagsimQuickstartTo check that DagSim is installed properly, run the following command in the console/terminal:dagsim-quickstartInstalling graphvizIf you usepip, you need to install graphviz on the system level in order to use the drawing functionality in DagSim.
Please follow the instrcutionshereon how to install graphviz depending on the
operating system.Simple examplePython codeSuppose we are interested in simulating two variables, X and Y, where X follows a standard Gaussian distribution, and Y
is the square of X.For each node we need a function to simulate the node's values:For X, we can use thenumpy.random.normalfunctionFor Y, we can use eithernumpy.poweror define our own function. We will use the second to illustrate how one can
use
user-define functions.# needed importsimportdagsim.baseasdsimportnumpyasnpHere, we define our ownsquarefunction:defsquare(arg):returnarg*argThen, we define the nodes in our graph/model by giving each node a name, the function to use in order to evaluate its
value, and the arguments of the function, if any:X=ds.Node(name="X",function=np.random.normal)Y=ds.Node(name="Y",function=square,kwargs={"arg":X})After that, we define the graph itself by giving it a name (optional) and a list of all the nodes to be included:graph=ds.Graph(name="demo_graph",list_nodes=[X,Y])If you wish, you can draw the graph by calling thedrawmethod, as follows:graph.draw()Finally, we simulate data from this graph by calling thesimulatemethod, and giving it the number of samples you
want to simulate, and a name for the csv_file (optional) where the data should be saved.data=graph.simulate(num_samples=10,csv_name="demo_data")Here,datawould be a dictionary with keys being the names of the nodes in the graph, and the corresponding values
being the simulated values for each node returned as a Pythonlist.For more detailed instructions, check
thispage, and
for other simple examples, please refer to thetutorialsfolder.YAML Specificationdagsim also allows the specification of a simulation using a YAML file. You can run dagsim on a YAML file by running:dagsimpath/to/yaml/file[-v|--verbose][-d|--draw][-ooutput/path|--output_path=output/path]For a tutorial on using a YAMl file for simulation, check
thispage.
Package: dagster-cloud
Dagster CloudThe enterprise orchestration platform that puts developer experience first, with fully serverless or hybrid deployments, native branching, and out-of-the-box CI/CD.Getting StartedThe complete Dagster Cloud feature set can be found on ourwebsite. You
cansign up for Dagster Cloudand get started with a free 30 day
trial.InstallationThe Dagster Cloud Agent is available on PyPI and officially supports Python 3.8.pip install dagster-cloudDocumentationYou can find the full Dagster Cloud documentation on ourwebsite.CommunityConnect with thousands of other data practitioners building with Dagster. Share knowledge, get help,
and contribute to the open-source project. To see featured material and upcoming events, check out
ourDagster Communitypage.Join our community here:🌟Star us on Github📥Subscribe to our Newsletter🐦Follow us on Twitter🕴️Follow us on LinkedIn📺Subscribe to our YouTube channel📚Read our blog posts👋Join us on Slack🗃Browse Slack archives✏️Start a Github Discussion
Package: dagster-custom
DagsterDagster is a data orchestrator for machine learning, analytics, and ETL.Dagster lets you define pipelines in terms of the data flow between reusable, logical components,
then test locally and run anywhere. With a unified view of pipelines and the assets they produce,
Dagster can schedule and orchestrate Pandas, Spark, SQL, or anything else that Python can invoke.Dagster is designed for data platform engineers, data engineers, and full-stack data scientists.
Building a data platform with Dagster makes your stakeholders more independent and your systems
more robust. Developing data pipelines with Dagster makes testing easier and deploying faster.
Package: dagster-ext
Meltano Dagster ExtentionThis project is still a work in progress. Please create an issue if you find any bugs.FeaturesLoad all Meltano jobs as Dagster jobs.Add all correspondig schedules to these jobs.(todo) Load all DBT models as Dagster assets.(todo) Load all Singer tap streams as Dagster assets.(todo) Ops to perform all Meltano actions.(todo) Extract Singer metrics from logs and store them using Dagster.Installation# Add the dagster-ext to your Meltano projectmeltanoaddutilitydagster# Initialize your Dagster projectmeltanoinvokedagster:initialize# Start Dagitmeltanoinvokedagster:startCommandsmeltanoinvokedagster:initializeSetup a new Dagster project and automatically load jobs and assets from your Meltano project.meltanoinvokedagster:startStart Dagit to serve your local Dagster deployment.Code ExamplesBelow are some code examples how to use thedagster-meltanopackage.Automatically load all jobs and schedules from your Meltano project.fromdagsterimportrepositoryfromdagster_meltanoimportload_jobs_from_meltano_projectmeltano_jobs=load_jobs_from_meltano_project("<path-to-meltano-root>")@repositorydefrepository():return[meltano_jobs]Install all Meltano pluginsfromdagsterimportrepository,jobfromdagster_meltanoimportmeltano_resource,meltano_install_op@job(resource_defs={"meltano":meltano_resource})definstall_job():meltano_install_op()@repository()defrepository():return[install_job]Create an arbitrary Meltano run commandfromdagsterimportrepository,jobfromdagster_meltanoimportmeltano_resource,meltano_run_op@job(resource_defs={"meltano":meltano_resource})defmeltano_run_job():tap_done=meltano_run_op("tap-1 target-1")()meltano_run_op("tap-2 target-2")(tap_done)@repository()defrepository():return[meltano_run_job]
Package: dagster-faculty
Python library for using Dagster in Faculty platform.Release StepsTo test pypi after applying your changes, runrelease-testpypijob from the Gitlab CI pipelineNote that if you get the errorFile already existsin the Pipeline, this is likely because of
theuse_scm_version.local_schemevalue insetup.py. We believe this will be suitably rare
enough to not need a workaround at present.Verify the new test release version onhttps://test.pypi.org/project/dagster-faculty/Once you have verified the test release, create an annotated tag on master, and then push it.git tag-a<version>git push origin--tagsVerify the new production release version onhttps://pypi.org/project/dagster-faculty/
Package: dagster-hashicorp
dagster-hashicorpA package for integratingHashicorp VaultwithDagster.RequirementsDagster 0.14+InstallationUse your favorite Python package manager to install the app from PyPI, e.g.pipinstalldagster-hashicorpUsage NotesVaultAuth methodTokenUser / PasswordApproleKubernetesExamplefromdagsterimportbuild_op_context,job,opfromdagster_hashicorp.vaultimportvault_resource@op(required_resource_keys={"vault"})defexample_vault_op(context):# Readsecret_data=context.resources.vault.read_secret(secret_path="secret/data/foo/bar")context.log.debug(f"Secret:{secret_data}")# Writecontext.resources.vault.create_or_update_secret(secret_path="secret/data/foo/bar",{"bar":"foo"})@job(resource_defs={"vault":vault_resource})defexample_job():example_vault_op()example_job.execute_in_process(run_config={"resources":{"vault":{"config":{"url":"vault-host:8200","auth_type":{"token":{"token":"s.your-token"}},}}}})# OR use environment variablesexample_job.execute_in_process(run_config={"resources":{"vault":{"config":{"url":"vault-host:8200","auth_type":{"token":{"token":{"env":"VAULT_TOKEN"}}},# or# "auth_type": {"userpass": {"username": {"env": "VAULT_USER"}, "password": {"env": "VAULT_PASS"}}},# or# "auth_type": {"approle": {"role_id": {"env": "VAULT_ROLE_ID"}, "secret_id": {"env": "VAULT_SECRET_ID"}}},# or# "auth_type": {"kubernetes": {"role": {"env": "VAULT_ROLE"}, "jwt_path": {"env": "VAULT_JWT_PATH"}}},}}}})
Package: dagster-hex
Hex Dagster LibraryInstallationTo install the library, use pip alongside your existing Dagster environment.pipinstalldagster-hexConfigurationFirst, you'll need to specify your Hex API Token key as a resource.# resources.pyfromdagster_hex.resourcesimporthex_resourceimportosAPI_KEY=os.getenv['DAGSTER_PROD_API']my_resource=hex_resource.configured({'api_key':API_KEY})OpsThehex_project_opwill call the Hex API to run a project until it completes.fromdagster_hex.resourcesimporthex_resourcefromdagsterimportjobfromdagster_hex.opsimporthex_project_opAPI_KEY='abc123'PROJ_ID='i-love-uuids'my_resource=hex_resource.configured({'api_key':API_KEY})run_hex_op=hex_project_op.configured({"project_id":PROJ_ID},name='run_job')@job(resource_defs={"hex":my_resource})defhex_job():run_hex_op()Asset MaterializationsOps will return anAssetMaterializationwith the following keys:run_url	
run_status_url	
trace_id	
run_id	
elapsed_time
Package: dagster-hightouch
Dagster-HightouchA Dagster library for triggering syncs in Hightouch.InstallationTo install the library, use pip alongside your existing Dagster environment.pipinstalldagster_hightouchConfigurationFirst, you'll need to specify yourHightouch APIkey as a resource.# resources.pyfromdagster_hightouch.resourcesimportht_resourceashightouch_resourceht_resource=hightouch_resource.configured({"api_key":"555555-4444-3333-2222-1111111111"},)OpsThehightouch_sync_opwill call the Hightouch API to trigger
a sync and monitor it until it completes.fromdagsterimportScheduleDefinition,get_dagster_logger,jobfromdagster_hightouch.opsimporthightouch_sync_opfrom.resourcesimportht_resource# Sync IDs are set as constants. You can also use# the sync slug, read the documentation for other# options.HT_WS="23620"HT_ORG="39619"# We define two configured sync opsrun_ht_sync_workspaces=hightouch_sync_op.configured({"sync_id":HT_WS},name="hightouch_sfdc_workspaces")run_ht_sync_orgs=hightouch_sync_op.configured({"sync_id":HT_ORG},name="hightouch_sfdc_organizations")# And create a job with the defined resources, specifying the dependencies.@job(resource_defs={"hightouch":ht_resource,})defht_sfdc_job():ht_orgs=run_ht_sync_orgs(start_after=ht_contacts)run_ht_sync_workspaces(start_after=ht_orgs)# And we schedule it to run every 30 mins.every_30_schedule=ScheduleDefinition(job=ht_sfdc_job,cron_schedule="*/30 * * * *")
Package: dagster-insights
Pre-release patches for dagster_insightsThis package contains pre-release fixes and optimizations for the officialdagster_cloud.dagster_insightsmodule included in thedagster-cloudpackage. To use the official version, runpip install dagster-cloudand import it as follows:from dagster_cloud import dagster_insightsTo enable the pre-release patches from this package, runpip install dagster-insightsand import this module before importingdagster_cloud.dagster_insights:import dagster_insights_prerelease  # Patches dagster_cloud.dagster_insights
from dagster_cloud import dagster_insightsThis allows testing against a pre-release version ofdagster_insightsand provides a faster turnaround time for fixes and optimizations. All reliable fixes are eventually merged intodagster_cloud.dagster_insightsand available in the official release.
Package: dagster-meltano
Dagster-meltanoA dagster plugin that allows you to run Meltano using Dagster.InstallationYou can install usingpip install dagster-meltano.ExamplesAn example of automatically loading all jobs and schedules from your Meltano project.fromdagsterimportrepositoryfromdagster_meltanoimportload_jobs_from_meltano_projectmeltano_jobs=load_jobs_from_meltano_project("<path-to-meltano-root>")@repositorydefrepository():return[meltano_jobs]An example of running an abitrarymeltano runcommand.fromdagsterimportrepository,jobfromdagster_meltanoimportmeltano_resource,meltano_run_op@job(resource_defs={"meltano":meltano_resource})defmeltano_run_job():tap_done=meltano_run_op("tap-1 target-1")()meltano_run_op("tap-2 target-2")(tap_done)@repository()defrepository():return[meltano_run_job]You can inject Meltano config with the following Dagster config.ops:tap_smoke_test_target_jsonl:config:env:TAP_SMOKE_TEST_STREAMS:'[{"stream_name":"new-stream","input_filename":"demo.json"}]'An example of running an arbitrary Meltano command.fromdagsterimportrepository,jobfromdagster_meltanoimportmeltano_resource,meltano_command_op@job(resource_defs={"meltano":meltano_resource})defmeltano_command_job():meltano_command_op("install loader tap-smoke-test")()@repository()defrepository():return[meltano_command_job]Development using VSCodeOpen this repository in Visual Studio Code.Install theRemote - Containersplugin for Visual Studio Code.Go to the example Meltano project rootcd meltano_projectInstall all pluginsmeltano installStart dagitmeltano invoke dagster:startVisitlocalhost:3000to access Dagit.
Package: dagster-pydantic
dagster-pydanticThis is a naive approach for using Pydantic types for Dagster Type Checking. The
packagedagster_pydanticincludes a really simple validation factory. Pydanitc
is not built as a data validation library, it's actually a great parser and an
OK validator in that order.The purpose of this validation step is to ensure the shape of the model is
correct before the data is consumed by the next op. Pydantic will stop you
during the instantiation of your model if the data is not up to spec but if you
want to check if the shape of the parsed data will match the model, that's where
the validation layer comes in.UsageclassMyPydanticModel(BaseModel):"""This is a Pydantic model."""a:intb:strMyPydanticModelDT=pydantic_to_dagster_type(MyPydanticModel)@op(out=Out(MyPydanticModelDT))defget_model():model=MyPydanticModel(a=1,b="hello")# This should fail type checking in the Dagit UI.model.b={}# type: ignorereturnmodelThis code will result in:Without this integration, you wouldn't see that typechecking error in the
console and the next op will consume adictin place of astr.DevelopingFirst, installJust$justinstallThis will run two poetry commands in your current terminal, one that configures
venvs to be in the project directory so the dependencies are accessable from
your local environment. The other will install the dependencies.$justshellThis opens a current shell into your poetry virtual env.Running tests$justtest# This only runs pytestDisclaimerI've only had my hands on Dagster for a few weeks. I love the declarative
framework, and the focus on flexibility + developer velocity. I am a bit iffy on
the pricing and what's offered in the "Standard plan", but otherwise I'm so
impressed by the open-source project.That being said, I'm a noobie. This is my first integration and also my first
Python module, so feel free to drop an issue if there's something I'm missing.
Package: dagster-scaleway
Dagster Scaleway (WIP)This repository contains aDagsterintegration forScaleway.It allows you to run Dagster pipelines onScaleway Serverless Jobs.Basic usageBuild a docker image containing your Dagster code and push it to the Scaleway Registry (or any other registry of your choice):dagsterprojectscaffold--namemy-dagster-projectcdmy-dagster-project

echo<<EOF > DockerfileFROM python:3.12-slim-bookwormWORKDIR /appCOPY . .RUN pip install .# Install the Dagster Scaleway module. You can also specify it in your "setup.py" fileRUN pip install dagster_scalewayEOFBuild and push the image:dockerbuild-trg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latest.
dockerpushrg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latestThen, configure thedagster.yamlfile to use this image:run_launcher:module:dagster_scalewayclass:ScalewayServerlessJobRunLauncherconfig:docker_image:rg.fr-par.scw.cloud/<your-namespace>/dagster-scaleway-example:latestRun Dagster locally:pipinstall-e".[dev]""dagster-scaleway"dagsterdevYour Dagster ops will be run as Scaleway Serverless Jobs! :tada:See theDagster documentationfor more information on how to get started with Dagster.ExamplesSee theexamplesfolder for examples of how to use this integration.
Package: dagster-serde
dagster-serdeType-aware ser/de library forDagster.@dataclassclassMyStruct:foo:strmy_struct=MyStruct(foo="bar")@asset(io_manager_key="json_io_manager",)defupstream()->MyStruct:returnmy_struct# my_struct is serialized to a json file (type hint is optional)@assetdefdownstream(upstream:MyStruct,):# my_struct is loaded as MyStruct based on the type hintassertupstream==my_structInstallationpipinstalldagster-serdeList of available extras:jsonyamlFor example, to use theJsonIOManager, installdagster-serde[json].IOManagersBaseSerdeUPathIOManagerBase class for IOManagers that ser/de to/from filesystems.Automatically deserializes inputs based on type annotations. If the input annotation is not a supported type, it's returned as a normal built-in type (e.g.str,int,dict, etc.). The supported types are:dataclasses.dataclassandserde.serde(usingpyserde). The full list of supported types can be foundhere.TODO: add support for other libraries likepydanticOptionaltype annotations are supported. If the input annotation isOptionaland is missing in the filesystem, theIOManagerwill skip loading the input and returnNoneinstead. If the output annotation isOptionaland the output isNone, theIOManagerwill skip writing the output to the filesystem.JsonIOManagerImplementsBaseSerdeUPathIOManagerfor JSON files. Usesorjson.YamlIOManagerImplementsBaseSerdeUPathIOManagerfor YAML files. Usespyyaml.ExamplesSeeexamples.DevelopmentInstallationpoetryinstall
poetryrunpre-commitinstallTestingpoetryrunpytest
Package: dagster-stitch
dagster-stitchThis library provides a Dagster integration for Stitch, a managed batch data ingestion service similar to Fivetran and Airbyte.DisclaimerPlease note this library is under active development and should be used cautiously! Currently it supports triggering replication jobs and materializing the resulting tables as Dagster assets that can be used in downstream jobs. Reconciliation and other features are not yet included but would be great additions.InstallationTo install the library, run:$pipinstalldagster-stitchFor development, it can be installed locally and tested with:$pipinstall-e.[lint,test]$pytestConfigurationSetupTo use the library, you must configure astitchresource in your Dagster instance. The resource requires anapi_keyto authenticate with Stitch. You can find or generate one in the Stitch UI underAccount Settings > API Access Keys.You will also need to note your Stitch account ID and the ID of the data source you want to replicate to be used in the asset or operation configuration. These can be found by navigating to the data source in the Stitch UI and looking at the URL. For example, if the URL ishttps://app.stitchdata.com/client/12345/pipeline/v2/sources/67890/summary, then the account ID is12345and the data source ID is67890.UsageHere's an example of how to instantiate an asset that will materialize thetable_nametable from thedata_source_iddata source:fromdagsterimportDefinitionsfromdagster_stitchimportstitch_resource,build_stitch_assetsstitch_instance=stitch_resource.configured({"api_key":"your_stitch_api_key","account_id":12345,})stitch_assets=build_stitch_assets(data_source_id=54321,destination_tables=["data_source_name.table_name"])definitions=Definitions(assets=stitch_assets,resources={"stitch":stitch_instance},)Note that you should include your data source name prefixed by a point in yourdestination_tablesand if the table do not correspond to a materialized asset, the materialization will raise an error.
Package: dagster-toolbox
Dagster ToolboxA set of tools to ease Dagster usageRequirementsPython 3.10+Dagster 1.1.5 / 0.17.5Installationpipinstalldagster-toolboxUsageNew object storage resourceDeclare resourcefromdagster_toolbox.resourcesimportObjectStorageclassAnalytics(ObjectStorage):bucket="analytics"@resourcedefanalytics_objects(init_context):returnAnalytics()RESOURCE_DEFS={"analytics_objects":analytics_objects,}Use resource@assetdefexport_files(context):partition_key=context.asset_partition_key_for_output()file_path=f"analytics/data/{partition_key}"file_resource=context.resource.analytics_objects.read(file_path)...
Package: dagstream
DagStreamDagStream is the Python package in order to manage relationship between functions, especially for data-preprocessing functions for machine learning applications. It offers simple API similar totaskflow.For more details, please refer to docs below.User ManualHow to installYou can intall dagstream by using pip.pip install dagstreamBasic UsageDefinition of DagDagStream class convert your functions into dag nodes.importdagstreamdeffuncA():print("funcA")deffuncB():print("funcB")deffuncC():print("funcC")deffuncD():print("funcD")deffuncE():print("funcE")deffuncF():print("funcF")stream=dagstream.DagStream()A,B,C,D,E,F=stream.emplace(funcA,funcB,funcC,funcD,funcE,funcF)A.precede(B,C)E.succeed(B,C,D)D.succeed(C)F.succeed(E)Execute whole dag functionsdagstream execute the functions based on relationship you defined.fromdagstream.executorimportStreamExecutor# construct functional dagfunctional_dag=stream.construct()executor=StreamExecutor(functional_dag)executor.run()In console, follwing items are shown.funcA
funcC
funcB
funcD
funcE
funcFDraw relationship of functions using MermaidYou can draw function dag relationship by usingMermaid.fromdagstream.viewersimportMermaidDrawerfunctional_dag=stream.construct()drawer.output(functional_dag,"path/to/output.mmd")The output is shown like below.stateDiagram
    direction LR
    state "funcE" as state_0
    state "funcB" as state_1
    state "funcF" as state_2
    state "funcD" as state_3
    state "funcC" as state_4
    state "funcA" as state_5
    state_0 --> state_2
    state_1 --> state_0
    state_2 --> [*]
    state_3 --> state_0
    state_4 --> state_0
    state_4 --> state_3
    [*] --> state_5
    state_5 --> state_4
    state_5 --> state_1
Package: dagtasets
## Install:Install dependencies`bash pip install--user--upgradenumpyscikit-imagescipyopencv-contrib-python`The package depends in a recent torchvision:`bash pip install--user--upgradegit+https://github.com/pytorch/vision`Install current version:`bash pip install--user--upgradegit+https://github.com/anguelos/dagtasets`## Contents:# dagtasetsA collection of datasets suited for documents and text image analysis.# scenethecizerA python module for generating document datasets for training models.# lm_utilGeneral utilities related to language models and training
Package: dagtools
Analyzing DAGs in PythonThedagtoolspackage provides utilities for computing various
graph-theoretic properties of DAGs (Directed Acyclic Graphs).Graph Formats and ConversionThedagtoolspackage is familiar with the below graph formats, and
can convert between any of them:DOT Formatnetworkx.Graphand related subclassesThe example below shows how to convert anetworkxGraph into a
DOT formatted string using thedagtools.convertmodule. Note, the inverse operation
is also available, via theconvert.got_to_graphfunction.>>>importnetworkx>>>fromdagtoolsimportconvert>>>g=networkx.MultiGraph()>>>g.add_nodes_from([1,2,3])>>>g.add_edges_from([(1,2),(1,3)],key=0)>>>print(convert.graph_to_dot(g))graph{1;2;3;1--2[key=0];1--3[key=0];}
Package: dagviz
DAGVIZDAGVIZ offers a "git log"-like visualization for DAGs represented in networkx.InstallationDAGVIZ can be installed withpip:pipinstalldagvizDocumentationDocumentation is available athttps://wimyedema.github.io/dagviz/.UsageDAGVIZ relies on networkx for the representation and manipulation of the
DAG. An SVG file can be generated as follows:importdagvizimportnetworkxasnx# Create the DAGG=nx.DiGraph()foriinrange(5):G.add_node(f"n{i}")G.add_edge(f"n0",f"n1")G.add_edge(f"n0",f"n2")G.add_edge(f"n0",f"n4")G.add_edge(f"n1",f"n3")G.add_edge(f"n2",f"n3")G.add_edge(f"n3",f"n4")# Create an SVG as a stringr=dagviz.render_svg(G)withopen("simple.svg","wt")asfs:fs.write(r)