Package: dagstd
Dagstd is a Python package containing a set of helper modules for use with
theDagsterdata orchestration tool.Dagster is a great tool, but there are occasions where you just need to pass in
a simple integer or string as input to a Dagster op, but in Dagster, inputs to
ops can only be outputs of other ops. This results in a lot of boilerplate
functions being written that just return a formatted string or even just an
integer. This is why Dagstd was created.FeaturesSimple ops for common numbersConstant value opsHelper ops for mathematical and string operationsOps for retrieving environment variablesSphinx autodoc support for Dagster opsUsageHere’s an example of a pure-Dagster graph that downloads a daily zip file and
extracts a known file name. Note: thedownload_large_fileop has been
omitted for brevity.importzipfilefromdatetimeimportdatetimefromdagsterimportop,job@opdefget_todays_date()->str:returndatetime.today().strftime()@opdeffive()->int:return5@opdefget_download_file_url(date:str)->str:returnf'https://example.com/{date}.csv'@opdefget_nth_file_name(n:int)->str:returnf'file_{n:02}.txt'@opdefextract_file_from_zip(context,zip_path:str,file_name:str)->str:withzipfile.ZipFile(zip_path)aszip_file:with(f'/tmp/{file_name}','wb')asf:f.write(zip_file.read(file_name))context.log.info(f'Extracted{file_name}from{zip_path}')returnf'/tmp/{file_name}'@jobdefprocess_data():date=get_todays_date()url=get_download_file_url(date)zip_path=download_large_file(url)file_name=get_nth_file_name(five())file_path=extract_file_from_zip(zip_path,file_name)And here’s the same graph, but with Dagstd ops.importzipfilefromdatetimeimportdatetimefromdagsterimportop,jobfromdagstd.constantsimportConstant,Fivefromdagstd.operationsimportfmt@opdefget_todays_date_string()->str:returndatetime.today().strftime("%Y-%m-%d")@opdefextract_file_from_zip(context,zip_path:str,file_name:str)->str:withzipfile.ZipFile(zip_path)aszip_file:with(f'/tmp/{file_name}','wb')asf:f.write(zip_file.read(file_name))context.log.info(f'Extracted{file_name}from{zip_path}')returnf'/tmp/{file_name}'@jobdefprocess_data():date=get_todays_date_string()url=fmt(Constant('https://example.com/{}.csv'),[date])zip_path=download_large_file(url)file_name=fmt(Constant('file_{}.txt'),[Five()])file_path=extract_file_from_zip(zip_path,file_name)This was just a small example, but it serves to show how much boilerplate can
be avoided when using Dagstd.Sphinx Autodoc PluginDagstd includes a Sphinx autodoc plugin that can be used to generate
documentation for Dagster ops. To use the autodoc plugin, add the following
to yourconf.pyfile:extensions=[...'dagstd.sphinx.parser',]By default, this will prefix all op documentation with(op). To change
this, add the following to yourconf.pyfile:dagstd_op_prefix='My Op'DocumentationDocumentation can be found athttps://dagstd.readthedocs.io/en/latest/readme.html.InstallationInstall Dagstd with pip:pipinstalldagstdDependenciesdagster >= 0.14.17ContributeI’m always looking for more ideas to add to Dagstd. If you have an idea, please
open an issue or pull request, or message me on GitHub.Issue Tracker:https://github.com/isaacharrisholt/dagstd/issuesSource Code:https://github.com/isaacharrisholt/dagstdSupportIf you are having issues, please let me know.LicenseThe project is licensed under the GNU GPLv3 license.