How to Share your Python Objects Across Different Environments in One Line of Code | by Khuyen Tran | Towards Data Science

Have you ever wanted to share Python objects between different scripts? Or to share findings between you and your teammates so that everybody could work on the most updated results? There are three difficulties you might encounter:
Your teammates may deploy a different environment from your script
The model that is trained by your teammates changes daily. So they need to reshare the model with you every day.
The files or models are large. Re-uploading them each time you deploy your script is cumbersome.‌
Wouldn’t it be easier if all you need to do is something like this to share the Dataframe like this
import datapane as dpdf = dp.Blob.get('profile', owner='khuyentran1401').download_df()df
or something like this to share your model?
import datapane as dppredictor = dp.Blob.get(name='predictor',owner='khuyentran1401').download_obj()X_TEST = [[10, 20, 30]]outcome = predictor.predict(X=X_TEST)
That is when we need Datapane’s Blob API
Datapane provides a collection of APIs which makes it easy to share Python analyses with other people. If you don’t know about Datapane already, I write an article about Datapane here.
towardsdatascience.com
Apart from APIs for creating reports and deploying your Notebook (shown in this other blog post... etc), Datapane also provides APIs for other common use-cases such as sharing blobs and managing secrets.
To illustrate the use of Blob, I use an example of a machine learning model from this article. Imagine you are training a model like this with good performance on the test set.
You want to send this model over to your teammate so that they can use your model to predict the new dataset. How can you do this?
You could set up a docker environment and send it over to your teammates. But if they just want to quickly test your model without the need of setting up the environment, datapane.Blob will be a better option.
Make sure to have the visibily='PUBLIC' if you want others to access it. Now others can use your blob for their code!
Outcome : [140.]Coefficients : [1. 2. 3.]`
Try to run the same code to see if you can access and use the predictor! Before copy and paste the code, make sure to sign up on Datapane to get your token beforehand. Then login on the terminal with your tokendatapane login — server=https://datapane.com/ — token=yourtoken
Now try
dp.Blob.get(name='predictor', owner='khuyentran1401').download_obj()
to see if you are able to access and use the predictor and produce the same results as above!
If you want to share your blob privately in your organization, follow the same process, but set the visibility of your blob to ORG
Besides uploading a Python object, you can also use blob to upload a Pandas Dataframe, a file.
And to get a Dataframe, file or object from your blob
For example, in the article, I wrote about my findings after scraping more than 1k Github profiles
towardsdatascience.com
To let you access to my dataset, all I need to do is to give you the name of the dataset and my Datapane’s account. Now you are ready to access it!
import datapane as dpdf = dp.Blob.get('profile', owner='khuyentran1401').download_df()df
Of course, to let you access to my data, I need to set the visibility of the blob to ‘PUBLIC’
Variables like database keys, passwords, or tokens should not be embedded in our code, especially when your script is visible to the outside world. Datapane’s Variable object provides a safe and secure way to create, store, and retrieve values that your scripts require.
For example, in this script, I used dp.Variable to store my GitHub token and my username so that when I share my code on Github, I don’t need to be worried about my token is exposed.
This variable is totally secret to me unless I specify it to be accessible by everyone with visibility='PUBLIC'.
If another person with a different account wants to access to my github_token variable and specify me as the owner, they will get an error
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url
because I am having my variable in private mode.
Another benefit: If I happen to reuse this variable in another script, I could easily access this token again without remembering what it is!
For those who are familiar to Python pickle, you might see the functionalities of Datapane’s Blob and Variable are similar to Python pickle. Yes they are almost similar but there are 3 main advantages ofBlob and Variable over pickle
With Blob, others don’t need to set up similar environments to yours in order to get the same model. For example, you may train a model on Sagemaker and want to use it in your script.
Instead of pickling and sending the pickle file to others, all you need to is to use upload to upload the change then your teammates use get to see the change!
You may accidentally make mistake in the new version and want to come back to the old version (i.e. train the wrong data). You can come back to the old version with version = in the parameters of get
Congratulations! You have just learned how to use Datapane’sBlob and Variable to share your Dataframe, file, Python object, and variable across different scripts or different environments.
Your teammates would be happy if all they need to do to see your results is one line of code without the hassle of setting up the environment. And you would be happy when your teammates could access the same data as you do.
Feel free to fork and play with the code for this article in this Github repo.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: