Deploy Python Code with Streamlit | Towards Data Science

I recently built and deployed a data application from my Mac using Python and did not write one line of HTML, CSS, or JavaScript — all made possible with a nifty package called Streamlit.
I recently deployed an early alpha version of a data dashboard with Steamlit. Although the app is in its infancy, I’m happy to have it in some kind of production state and I am excited to continue improving on it. Based on my experience, this article contains some of the biggest gotchas and tips on breaking through a few final issues that you may encounter with a Streamlit deployment. This article is not a full guide, for that, just check out the Streamlit site which does a good job at the step-by-step process.
For the longest time, I’ve been wondering about the best way to deploy my data analysis. For example, we do all this work and nug away at code, but there always seems to be a gap. It is almost as if there is a valley where projects go to die before we can share them.
There is some valley where projects go to die before we can share them.
Possible solutions include Jupyter notebooks and Deepnote which are great for sharing code and analysis. Another one is ObservableHQ which is pretty good at pushing D3 visuals out the door. There are dozens of other ways to get the job done as well. However, despite the options, I’ve never been able to connect with a deployment method with just the right features. Fortunately, my mentor gave me some advice to try a combination of Plotly and Streamlit.
Although I knew about Plotly, I never heard of Streamlit before. Billed as the “fastest way to build and share data apps,” Streamlit makes the promise of turning your code into a web app within minutes with zero work on the front-end. I thought it was too good to be true, but as it turns out, Streamlit works.
Free
Available via pip install
Good documentation
GitHub repo integration
Handles the front-end
Only requires Python
If this all sounds interesting, check out the Streamlit page. The documentation and instructions are pretty clear; however, there are a few important design features that I think are worth considering in your code. Read on for more.
Sometimes you have a ton of data, 500 thousand or 1 million rows of data — how best to deploy a data app that points to such a large dataset? Although there are plenty of answers, to include avoiding having a single massive table in the first place, let us proceed with solving this specific problem. Now what?
There are three main problems with handling a single, large data table. First, on disk, a file of this size may range in the hundreds of MBs. Second, if deploying with Streamlit, you have to use GitHub which means a hard limit of about 50 MBs. Third and lastly, loading a large dataset can crash the app. As a result, consider compressing DataFrames to compressed BZ2 pickle files instead of CSV files.
Big Data, Big Problems: 245 MB .csv file or 20 MB .bz2 file?
Inserted on 13 January 2020 — although this compression method works for large data as a means to get to production, as a next step, you should certainly take additional steps to reduce the size of each file as much as possible. More about that and optimization in a follow-up article linked below.
towardsdatascience.com
To handle large DataFrames, after reading the initial csv file, design your workflow to read and write pickled data with BZ2 compression. The BZ2-Pickle combination does good for you in two big ways. First, the pickled format retains the state of all your columns and takes up less disk space than CSV files. Second, BZ2 efficiently compresses data to tiny bytes along with pretty great decompression on the back end.
Fortunately, Pandas makes this an easy task. All you need is to have ‘.bz2’ as the file extension.
import pandas as pd# given a dataframe of 600,000 records..path = 'data/product_sales.bz2'df.to_pickle(path)# pandas writes compressed pickle .bz2 based on filename# to_pickle() has parameter of compression=infer# read back in to dataframedf = pd.read_pickle(path)
Like most projects, the workflow starts with data in CSV format; however, internal to the app, I designed the program to read and write DataFrames in pickle formats. A major reason to use pickled DataFrames is that the serialized file retains various metadata — if you set dtypes to strings, integers, or category, those dtypes are retained every time you read in the data. On the other hand, with CSV files, you have to re-process the data back to a suitable DataFrame.
Although pickle files are pretty sweet, during the last few steps of deployment with Streamlit, I ran into a brick wall of an error with the pickle protocol.
Apparently, the Pandas to_pickle() method defaults to a protocol of version 5 which is not universally supported. As a result, although a standard Pandas pickled DataFrame may work in testing on your local machine, deployment to a server is another story.
ValueError: unsupported pickle protocol: 5Traceback:File "/usr/local/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script    exec(code, module.__dict__)File "/app/app_courts/main.py", line 56, in <module>    run_app()File "/app/app_courts/main.py", line 52, in run_app    , classify=FalseFile "/app/app_courts/do_data/getter.py", line 124, in to_df    df = pd.read_pickle(path)File "/home/appuser/.local/lib/python3.7/site-packages/pandas/io/pickle.py", line 182, in read_pickle    return pickle.load(f)
When faced with an error, I sometimes go with an alternate solution, i.e., something works just as well to produce the same result. However, I had little choice with the data file because of various constraints. While searching for an answer, I discovered that the solution is fairly simple — change the Pandas to_pickle() protocol from default to version 2. When Pandas pickle is combined with BZ2 compression, the result is a super small, super convenient, and very compatible data file.
# to avoid pickle protocol error# change params from 5 to 2path = 'data/product_sales.bz2'df.to_pickle(path, protocol=2)
When Pandas pickle is combined with BZ2 compression, the result is a super small, super convenient, and very compatible data file.
In addition to visualizations, the app presents the user with a narrative of the dataset’s high-level numbers. For example, there are a few sentences that explain the counts and frequencies in a written format. To keep the reader in the flow of the narrative, I wanted to format numbers as strings with thousands separators instead of pure integers. One way to format numbers as text is with the Locale library.
# one way to format numbers as stringsimport localelocale.setlocale(locale.LC_ALL, 'en_US')case_id = 8456981formatted = locale.format_string("%d", case_id, grouping=True)# formatted -> '8,456,981'
Unfortunately, while this method works just fine on my local machine, the deployment drops dead with an unsupported locale setting error.
For whatever reason, locale throws a massive wrench into the deployment. Although there may be some other option to resolve locale settings, I realized there may be a better alternative.
Error: unsupported locale settingTraceback:File "/usr/local/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script    exec(code, module.__dict__)File "/app/app_courts/main.py", line 8, in <module>    from app.app import AppFile "/app/app_courts/app/app.py", line 6, in <module>    from analyze_data.charts import ChartsFile "/app/app_courts/analyze_data/charts.py", line 25, in <module>    locale.setlocale(locale.LC_ALL, 'en_US')File "/usr/local/lib/python3.7/locale.py", line 608, in setlocale    return _setlocale(category, locale)
The solution is always f string, a fantastic and nifty invention that will help you when you need help the most.
The f string is a fantastic invention. Although the syntax may seem intimidating (it takes some getting used to), it works. To return an integer formatted as a neat number with thousands separators try the following syntax.
case_id = 8456981formatted = f"{case_id:,d}"# formatted -> '8,456,981'
Managing file paths and directories are a pain. For example, the syntax might change depending on whether you are a Mac, a PC, in a virtual environment, or some other wild combination. To deal with the uncertainty and like many others, I developed a habit of coding file path separators with the OS library to ensure compatibility across different environments.
# an example of creating a cross-platform file pathimport osfolder = 'data'self.path = os.environ['PWD'] + os.sep + folder""" a go-to approach given a root and some sub-folders-root/app.py----analysis/----data/----figures/"""
For whatever reason, although the app works on my local machine, the online Streamlit deployment returns a KeyError when I try to access ‘PWD’ or print working directory.
Although I am not sure why os.environ[‘PWD’] fails to work, I ended up changing the approach to rely on relative paths.
KeyError:'PWD'Traceback:File "/usr/local/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script    exec(code, module.__dict__)File "/app/app_courts/main.py", line 12, in <module>    reader = Reader()File "/app/app_courts/do_data/getter.py", line 17, in __init__    self.path = os.environ['PWD'] + os.sep + folder + os.sepFile "/usr/local/lib/python3.7/os.py", line 681, in __getitem__    raise KeyError(key) from None
If Streamlit runs from the project root, i.e. main.py or app.py, then consider simply referencing sub-folders with a relative path instead. Consider the following set-up where streamlit runs app.py (from the root directory).
# from terminal, run app.py>>> streamlit run app.py""" project structure-root/app.py----analysis/----data/'product_sales.csv'----figures/"""
In this scenario, if the app needs to read a file from the data sub-folder, work with os.sep.join() with relative location in mind. While testing this solution, I noticed that this works just fine both on my local machine and in the deployment.
# an alternative to using OS with PWDpath = os.sep.join([self.root, filename])# path -> data/product_sales.csv
With Streamlit, it is possible to deploy a Python-based project as a web app with little to no front-end work. Although the process takes longer than just a few minutes, it is doable within a few hours or a day of work. I am sharing this article to pass along the word about Streamlit and to share a few tips on working through some issues.
As for the issues, I wish I had designed around these roadblocks from the start. Although each issue was fairly minor, each one prevented the app from deploying and required a significant amount of time to research and refactor. Hopefully, these notes on my mistakes can benefit your next project.
Thanks for reading. If you are considering deploying with Streamlit or are troubleshooting, I hope this article helps.
My GitHub Repo