Package: pydwt
pydwtThe pydwt library provides a set of tools for orchestrating tasks of data processing in a directed acyclic graph (DAG). This DAG is composed of tasks that have dependencies between them and can be executed in parallel or sequentially, depending on their dependencies.Installationpip install pydwtGuideIn this document, we will provide a brief explanation of the main modules of the pydwt library, which are:session.py: module for interacting with a database and creating DataFrame objects to manipulate data.dataframe.py: module for defining a DataFrame class for working with data.task.py: module for defining tasks in the DAG.dag.py: module for creating and traversing the DAG.workflow.py: module for running the DAG.SessionThesession.pymodule is responsible for interacting with a database and creating DataFrame objects to manipulate data. To use this module, you need to create an instance of the Session class, passing an SQLAlchemy engine object and the schema of the database (if it has one). Then, you can use the table method to create a DataFrame object from a table in the database.Here is an example of how to create a Session object and use the table method to create a DataFrame object:fromsqlalchemyimportcreate_enginefrompydwt.sql.sessionimportSessionengine=create_engine("postgresql://user:password@localhost/dbname")session=Session(engine,schema="my_schema")df=session.table("my_table")DataFrameThedataframe.pymodule defines a DataFrame class for working with data. A DataFrame object is essentially a table with labeled columns and rows. You can use it to perform operations such as selecting, filtering, grouping, and aggregating data.You can also materialize a DataFrame as a table or view in the database by calling the materialize method.Here is an example of how to create a DataFrame object and perform some operations on it:frompydwt.sql.sessionimportSessionfrompydwt.sql.dataframeimportDataFramesession=Session(engine,schema="my_schema")df=session.table("my_table")# select some columnsdf=df.select(df.col1,df.col2)# filter rows based on a conditiondf=df.where(df.col1>10)# group by a column and aggregate another columndf=df.group_by(df.col2,agg={df.col1:(func.sum,"sum_col1")})# show the resulting DataFramedf.materialize("new_table",as_="table")TaskThetask.pymodule defines a Task class for representing a task in the DAG. A Task object has a run method that is responsible for executing the task. You can also define the task's dependencies, schedule, and other parameters when creating the object.To create a Task object, you can use the@Taskdecorator and define the run method. Here is an example of how to create a Task object:frompydwt.core.taskimportTask@Task()deftask_one():df=session.table("features")df=df.with_column("new_column",case((df.preds=="hw","W")))df.materialize("new_table",as_="table")@Task(depends_on=[task_one])deftask_two():df=session.table("new_table")df=df.where((df.new_column=="W"))df=df.with_column("new_column",case((df.preds=="hw","W")))df.show()Create a new pydwt project:pydwt new <my_project>This command will create a new project with the name "my_project" and the required file structure.my_project/
    models/
        example.py
    dags/
settings.ymlproject_name/models: where you will put your tasksproject_name/dags/: where the corresponding dag PNG file will besettings.yml: a configuration file for your project. This file includes the configuration options for your project, such as the path to your data directory.Export the DAGpydwt export-dagwill export the current state of your dag in theproject_name/dags/as PNG file with timestamp.Run your projectpydwt run <module.function_name>If no argument provided will run the current state of your DAG. It will process the tasks in the DAG by level and parallelize
it with theThreadExecutor. It a task failed then its child tasks will not be run.If argument provided in the form ofmodule.function_namefor instanceexample.task_onethen will run all tasks in the dag leading to this task.If parent tasks succeeded then run the task.Test your connection setuppydwt test-connectionwill test the current setup of your DB connectiona according to yoursettings.ymlfile.Configuration of your pydwt projectThesettings.ymlfile is a configuration file for your pydwt project. It stores various settings such as the project name, database connection details, and DAG tasks.connectionThe connection section contains the configuration details for connecting to the database. The available options are:url: the connection string to your dbYou can add others keys that will be forwarded to the underlyingcreate_enginefunction
for instance you can add aecho : trueand it will callcreate_engine(url=url, echo=echo)seeheresupported args.projectThe project section contains the project-related settings. The available options are:name: the name of the projecttasksThis section contains the configuration for each task defined in the pydwt project.Each task is identified by its name, and the configuration is stored as a dictionary.The dictionary can contain any key-value pairs that the task implementation may need to use, but it must have a key namedmaterialize.Thematerializekey specifies how the task output should be stored. The value can be eitherviewortable.
The value of the materialize key determines whether the task output should be stored as a SQL view or a SQL table. If the value is view, the output is stored as a SQL view. If the value is table, the output is stored as a SQL table.Each task implementation can access its configuration by injecting the config argument and specifying Provide[Container.config.tasks.<task_name>]. The injected config argument is a dictionary containing the configuration for the specified task.example :frompydwt.core.taskimportTaskfromdependency_injector.wiringimportinject,Providefrompydwt.core.containersimportContainer@Task()@injectdeftask_one(config:dict=Provide[Container.config.tasks.task_one]):print(config)@Task(depends_on=[task_one])deftask_two():print("somme processing")sourcesThe sources section contains the database sources that can be used in the project. Each source must have a unique name and specify the schema and table to use for the source.sources:name_alias:table:xxxschema:yyyYou can then required this datasource in your tasksfrompydwt.core.taskimportTaskfromdependency_injector.wiringimportinject,Providefrompydwt.core.containersimportContainer@Task()@injectdeftask_one(config:dict=Provide[Container.config.tasks.task_one],repo=Provide[Container.datasources],):df=repo.get_sources("name_alias")LicenseThis project is licensed under GPL.