Package: viadot
ViadotDocumentation:https://dyvenia.github.io/viadot/Source Code:https://github.com/dyvenia/viadotA simple data ingestion library to guide data flows from some places to other places.Getting Data from a SourceViadot supports several API and RDBMS sources, private and public. Currently, we support the UK Carbon Intensity public API and base the examples on it.fromviadot.sources.uk_carbon_intensityimportUKCarbonIntensityukci=UKCarbonIntensity()ukci.query("/intensity")df=ukci.to_df()dfOutput:fromtoforecastactualindex02021-08-10T11:00Z2021-08-10T11:30Z211216moderateThe abovedfis a python pandasDataFrameobject. The above df contains data downloaded from viadot from the Carbon Intensity UK API.Loading Data to a SourceDepending on the source, viadot provides different methods of uploading data. For instance, for SQL sources, this would be bulk inserts. For data lake sources, it would be a file upload. We also provide ready-made pipelines including data validation steps using Great Expectations.An example of loading data into SQLite from a pandasDataFrameusing theSQLiteInsertPrefect task:fromviadot.tasksimportSQLiteInsertinsert_task=SQLiteInsert()insert_task.run(table_name=TABLE_NAME,dtypes=dtypes,db_path=database_path,df=df,if_exists="replace")Set upNote: If you're running on Unix, after cloning the repo, you may need to grant executable privileges to theupdate.shandrun.shscripts:sudo chmod +x viadot/docker/update.sh && \
sudo chmod +x viadot/docker/run.sha) userClone themainbranch, enter thedockerfolder, and set up the environment:git clone https://github.com/dyvenia/viadot.git && \
cd viadot/docker && \
./update.shRun the enviroment:./run.shb) developerClone thedevbranch, enter thedockerfolder, and set up the environment:git clone -b dev https://github.com/dyvenia/viadot.git && \
cd viadot/docker && \
./update.sh -t devRun the enviroment:./run.sh -t devInstall the library in development mode (repeat for theviadot_jupyter_labcontainer if needed):docker exec -it viadot_testing pip install -e . --userRunning testsTo run tests, log into the container and run pytest:docker exec -it viadot_testing bash
pytestRunning flows locallyYou can run the example flows from the terminal:docker exec -it viadot_testing bash
FLOW_NAME=hello_world; python -m viadot.examples.$FLOW_NAMEHowever, when developing, the easiest way is to use the provided Jupyter Lab container available in the browser athttp://localhost:9000/.Executing Spark jobsSetting upTo begin using Spark, you must first declare the environmental variables as follows:DATABRICKS_HOST = os.getenv("DATABRICKS_HOST")
DATABRICKS_API_TOKEN = os.getenv("DATABRICKS_API_TOKEN")
DATABRICKS_ORG_ID = os.getenv("DATABRICKS_ORG_ID")
DATABRICKS_PORT = os.getenv("DATABRICKS_PORT")
DATABRICKS_CLUSTER_ID = os.getenv("DATABRICKS_CLUSTER_ID")Alternatively, you can also create a file called.databricks-connectin the root directory of viadot and add the required variables there. It should follow the following format:{
  "host": "",
  "token": "",
  "cluster_id": "",
  "org_id": "",
  "port": ""
}To retrieve the values, follow step 2 in thislinkExecuting Spark functionsTo begin using Spark, you must first create a Spark Session:spark = SparkSession.builder.appName('session_name').getOrCreate().sparkwill be used to access all the Spark methods. Here is a list of commonly used Spark methods (WIP):spark.createDataFrame(df): Create a Spark DataFrame from a Pandas DataFramesparkdf.write.saveAsTable("schema.table"): Takes a Spark DataFrame and saves it as a table in Databricks.Ensure to use the correct schema, as it should be created and specified by the administratortable = spark.sql("select * from schema.table"): example of a simple query ran through PythonHow to contributeFork repository if you do not have write accessSet up locallyTest your changes withpytestSubmit a PR. The PR should contain the following:new/changed functionalitytests for the changeschanges added toCHANGELOG.mdany other relevant resources updated (esp.viadot/docs)The general flow of working for this repository in case of forking:Pull before making any changesCreate a new branch withgit checkout -b <name>Make some work on repositoryStage changes withgit add <files>Commit the changes withgit commit -m <message>Note: See out Style Guidelines for more information about commit messages and PR namesFetch and pull the changes that could happen while working withgit fetch <remote> <branch>
git checkout <remote>/<branch>Push your changes on repostory usinggit push origin <name>Use merge to finish your push to repositorygit checkout <where_merging_to>
git merge <branch_to_merge>Please follow the standards and best practices used within the library (eg. when adding tasks, see how other tasks are constructed, etc.). For any questions, please reach out to us here on GitHub.Style guidelinesthe code should be formatted with Black using default settings (easiest way is to use the VSCode extension)commit messages should:begin with an emojistart with one of the following verbs, capitalized, immediately after the summary emoji: "Added", "Updated", "Removed", "Fixed", "Renamed", and, sporadically, other ones, such as "Upgraded", "Downgraded", or whatever you find relevant for your particular situationcontain a useful description of what the commit is doingSet up Black for development in VSCodeYour code should be formatted with Black when you want to contribute. To set up Black in Visual Studio Code follow instructions below.Installblackin your environment by writing in the terminal:pip install blackGo to the settings - gear icon in the bottom left corner and selectSettingsor type "Ctrl" + ",".Find theFormat On Savesetting - check the box.Find thePython Formatting Providerand select "black" in the drop-down list.Your code should auto format on save now.