Repository: streamlit/streamlit

Issue:
Speed up Streamlit with gzip (or equivalent) compression for big packets
# Problem

Sending a big DataFrame (or any big packet) takes a long time. This was a [problem for previous users](https://github.com/streamlit/streamlit-old-private/issues/1237), and it's [coming up again in the forums](https://discuss.streamlit.io/t/scalabilty-of-streamlit-for-pandas/131/3?u=adrien_treuille).

As part of our push for [Streamlit Performance Improvements](https://docs.google.com/document/d/1enxtZeqESJGbV-zgR4YpCByAPyNLk9yDaN7Lf2GbxKg/edit#) we should address this!

# Solution

At a certain threshold (or maybe always) compress the packet with gzip (or equivalent).

## Side Note

Would using [Apache Arrow](https://github.com/streamlit/streamlit-old-private/issues/607) also give us floating point compression??


Discussion:
I work with large sets of textual data which is highly compressible and this would be really useful.

Assuming you are planing to decompress in javascript and not rely on browser built in compression, may I suggest using zstandard instead of gzip? It compresses both better than gzip and decompresses at a much higher speed (see https://facebook.github.io/zstd/ for comparison)

More importantly though, zstandard has a much wider range of speed-compression tradeoff and it would be great to have this configurable so users can tune it to the kind of data they have.

There are both python and javascript implementations which are straight forward to use:
https://pypi.org/project/zstandard/
https://github.com/yoshihitoh/zstd-codec

For extremely large datasets (Gigabytes+) there are some challenges that the solution should keep in mind that I ran into with text processing - namely that some compression algorithms (like gzip) require the whole file to decompress it because of the way it stores the compression.  This has some advantages in compression size, but the tradeoff is that if you don't have the whole file you can't see what's in it (Or sample sections of it if you're searching for time series data for example).  If initially this supports gzip, that would be great, but for the datasets I've had to deal with regularly, (again, specifically with time series data in my case), the ability to "chunk" pieces of the file without having the entire thing is advantageous for performance/memory use.

EDIT: clarification - it's not that it needs the "whole file" to decompress in a stream, but it forces you to go through the file sequentially rather than being able to sample in different places - so if I were searching for something in my text file that had predicable structure (Not always the case), I would potentially need to read the entire file to get the timeframe I'm looking for, because I can't see the last block of the file without opening all of the ones previous to it.
@bubthegreat This issue is about using compression internally between the python and browser components of streamlit and won't affect how you use your own data.

Resolving patch:
diff --git a/lib/streamlit/server/Server.py b/lib/streamlit/server/Server.py
--- a/lib/streamlit/server/Server.py
+++ b/lib/streamlit/server/Server.py
@@ -585,6 +585,16 @@ def on_close(self):
         self._server._close_report_session(self._session.id)
         self._session = None
 
+    def get_compression_options(self):
+        """Enable WebSocket compression.
+
+        By default, this method returns None, which means compression
+        is disabled. Returning an empty dict enables it.
+
+        (See the docstring in the parent class.)
+        """
+        return {}
+
     @tornado.gen.coroutine
     def on_message(self, payload):
         if not self._session:


Test patch:
diff --git a/lib/tests/streamlit/Server_test.py b/lib/tests/streamlit/Server_test.py
--- a/lib/tests/streamlit/Server_test.py
+++ b/lib/tests/streamlit/Server_test.py
@@ -147,6 +147,21 @@ def test_multiple_connections(self):
             yield gen.sleep(0.1)
             self.assertFalse(self.server.browser_is_connected)
 
+    @tornado.testing.gen_test
+    def test_websocket_compression(self):
+        with self._patch_report_session():
+            yield self.start_server_loop()
+
+            # Connect to the server, and explicitly request compression.
+            ws_client = yield tornado.websocket.websocket_connect(
+                self.get_ws_url("/stream"), compression_options={}
+            )
+
+            # Ensure that the "permessage-deflate" extension is returned
+            # from the server.
+            extensions = ws_client.headers.get("Sec-Websocket-Extensions")
+            self.assertIn("permessage-deflate", extensions)
+
     @tornado.testing.gen_test
     def test_forwardmsg_hashing(self):
         """Test that outgoing ForwardMsgs contain hashes."""