Package: sf-hamilton-contrib
Off-the-shelf Hamilton DataflowsHere you'll find documentation about the sf-hamilton-contrib package that curates a collection of Hamilton Dataflows that are
ready to be used in your own projects. They are user-contributed and maintained, with
the goal of making it easier for you to get started with Hamilton.We expect this collection to grow over time, so check back often! As dataflows become mature we
will move them into the official sub-package of the respository and become maintained by the
Hamilton team.UsageThere are two ways to get access to dataflows in this package. For either approach,
the assumption is that you have the requisite python dependencies installed on your system.
You'll get import errors if you don't. Don't know what you need, we have convenience functions to help!Static installationThis approach relies on you installing the package on your system. This is the recommended path for
production purposes as you can version-lock your dependencies.To install the package, run:pipinstallsf-hamilton-contrib--upgradeOnce installed, you can import the dataflows as follows.Things you need to know:Whether it's a user or official DAGWorks supported dataflow. If user, what the name of the user is.The name of the dataflow.fromhamiltonimportdriver# from hamilton.contrib.dagworks import NAME_OF_DATAFLOWfromhamilton.contrib.user.NAME_OF_USERimportNAME_OF_DATAFLOWdr=(driver.Builder().with_config({})# replace with configuration as appropriate.with_modules(NAME_OF_DATAFLOW).build())# execute the dataflow, specifying what you want back. Will return a dictionary.result=dr.execute([NAME_OF_DATAFLOW.FUNCTION_NAME,...],# this specifies what you want backinputs={...}# pass in inputs as appropriate)To find an examplego to the hub.Dynamic installationHere we dynamically download the dataflow from the internet and execute it. This is useful for quickly
iterating in a notebook and pulling in just the dataflow you need.fromhamiltonimportdataflows,driver# downloads into ~/.hamilton/dataflows and loads the module -- WARNING: ensure you know what code you're importing!# NAME_OF_DATAFLOW = dataflows.import_module("NAME_OF_DATAFLOW") # if using official DAGWorks dataflowNAME_OF_DATAFLOW=dataflows.import_module("NAME_OF_DATAFLOW","NAME_OF_USER")dr=(driver.Builder().with_config({})# replace with configuration as appropriate.with_modules(NAME_OF_DATAFLOW).build())# execute the dataflow, specifying what you want back. Will return a dictionary.result=dr.execute([NAME_OF_DATAFLOW.FUNCTION_NAME,...],# this specifies what you want backinputs={...}# pass in inputs as appropriate)To find an examplego to the hub.ModificationGetting started is one thing, but then modifying to your needs is another. So we have a prescribed
flow to enable you to take a dataflow, and copy the code to a place of your choosing. This allows
you to easily modify the dataflow as you see fit.Run this in a notebook or python script to copy the dataflow to a directory of your choosing.fromhamiltonimportdataflows# dynamically pull and then copyNAME_OF_DATAFLOW=dataflows.import_module("NAME_OF_DATAFLOW","NAME_OF_USER")dataflows.copy(NAME_OF_DATAFLOW,destination_path="PATH_TO_DIRECTORY")# copy from the installed libraryfromhamilton.contrib.user.NAME_OF_USERimportNAME_OF_DATAFLOWdataflows.copy(NAME_OF_DATAFLOW,destination_path="PATH_TO_DIRECTORY")You can then modify/import the code as you see fit. Seecopy()for more details.How to contributeIf you have a dataflow that you would like to share with the community, please submit a pull request
to this repository. We will review your dataflow and if it meets our standards we will add it to the
package. To submit a pull request please usethis template. To access it, create a new Pull Request, then hit thepreviewtab, and click the link to appendtemplate=HAMILTON_CONTRIB_PR_TEMPLATE.mdto the URL.Dataflow standardsWe want to ensure that the dataflows in this package are of high quality and are easy to use. To that end,
we have a set of standards that we expect all dataflows to meet. If you have any questions, please reach out.Standards:The dataflow must be a valid Python module.It must not do anything malicious.It must be well documented.It must work.It must follow our standard structure as outlined below.Getting started with developmentTo get started with development, you'll want to first fork the hamilton repository from the github UI.Then, clone it locally and install the package in editable mode, ensuring you install any dependencies required for the initilization scriptcdhamilton# Your forkpipinstall-e"./contrib[contribute]"# Note that this package lives under the `contrib` folderNext, you need to initialize your dataflow. This will create the necessary files and directories for you to get started.init-dataflow-u<your_github_username>-n<name_of_dataflow>This will do the following:Create a package undercontrib/hamilton/contrib/user/<your_github_username>with the appropriate files to describe youauthor.md-- this will describe you with links out to github/socials__init__.py-- this will be an empty file that allows you to import your dataflowCreate a package undercontrib/hamilton/contrib/user/<your_github_username>/<name_of_dataflow>with the appropriate files to describe your dataflow:README.mdto describe the dataflow with the standard headings__init__.pyto contain the Hamilton coderequirements.txtto contain the required packages outside of Hamiltontags.jsonto curate your dataflowvalid_configs.jsonlto specify the valid configurations for it to be rundag.pngto show one possible configuration of your dataflowAdd all the above files to git!These are all required. You do not have to use the initialization script -- you can always copy the files over directly. That said, it is idempotent (it will fill out any missing files),
and will ensure that you have the correct structure.Developing your dataflowTo get started, you'll want to do the following:Fill out your__init__.pywith the appropriate code -- seethis issueif you want some inspiration for where to get startedFill out the sections of yourREADME.mdwith the appropriate documentation -- follow one of the approved dataflowsFill out yourtags.jsonwith the appropriate tags -- follow one of the approved dataflowsFill out yourvalid_configs.jsonlwith the appropriate configurations -- this is not necessary if you have no configurations that can change the shape of your DAGGenerate a visual representation of your DAG -- you can use the followingif __name__ == '__main__'block to do so:import__init__asmy_modulefromhamiltonimportbase,driverdr=driver.Driver({},my_module,adapter=base.DefaultAdapter(),)# create the DAG imagedr.display_all_functions("dag",{"format":"png","view":False})Push a branch back to your forkOpen up a pull request to the main Hamilton repo!Commit the files we just addedCreate a PRTag one of the maintainerselijahbenizzy,skrawcz, orziltofor a reviewPing us onslackif you don't hear back within a few daysUsername ManagementAs usernames map to packages, we need to ensure that they are valid. To that end, we have a few rules:If your username contains hyphens, replace them with underscores.If your username starts with a number, prefix it with an underscore.If your author name is a python reserved keyword. Reach out to the maintainers for help.If the above apply, run theinit-dataflowcommand with-sto specify a sanitized username.Got questions?Join ourslackcommunity to chat/ask Qs/etc.