Repository: streamlit/streamlit

Issue:
st.write throws value is null when string is too big
# Summary
calling `st.write` with a string that is too big will result on an error message on the front end with message "value is null"

<img width="551" alt="Screen Shot 2019-10-16 at 3 57 57 PM" src="https://user-images.githubusercontent.com/934511/66950384-a17d0e00-f02e-11e9-87bb-1df7158794b5.png">

# Steps to reproduce
1. create a script with a variable that holds a string that's more than weights more than 50mb
2. streamlit run yourscript.py
3. message will appear on the front end


## Expected behavior:
The error message should be clearer.

## Actual behavior:
Error message does not point to the string length constrain of `st.write`.




Discussion:
This occurs with video files too, with `st.video`
It's occurring with st.dataframe as well when the dataframe is big.
Running into the same problem with `st.video()`, will try to reproduce later.
Confirmed: `st.write` issue still outstanding.

#415 (merged but not yet released) will clear up the st.video issue.

#1021 (in progress) might clear up the dataframe issue.
Hi All, I have, from what I gather, the same problem loading large Datasets for visualisation (see also https://discuss.streamlit.io/t/very-large-datasets/3168?u=brey).

Any possibility to fix it soon?
Hi @brey -- Loading large datasets isn't exactly the same issue.  We're still working on a major overhaul of the dataframe processing code.  See #1021 for updates.

Resolving patch:
diff --git a/lib/streamlit/server/server_util.py b/lib/streamlit/server/server_util.py
--- a/lib/streamlit/server/server_util.py
+++ b/lib/streamlit/server/server_util.py
@@ -68,22 +68,19 @@ def serialize_forward_msg(msg):
     msg_str = msg.SerializeToString()
 
     if len(msg_str) > MESSAGE_SIZE_LIMIT:
-        _convert_msg_to_exception_msg(msg, RuntimeError("Data too large"))
+        import streamlit.elements.exception_proto as exception_proto
+
+        error = RuntimeError(
+            f"Data of size {len(msg_str)/1e6:.1f}MB exceeds write limit of {MESSAGE_SIZE_LIMIT/1e6}MB"
+        )
+        # Overwrite the offending ForwardMsg.delta with an error to display.
+        # This assumes that the size limit wasn't exceeded due to metadata.
+        exception_proto.marshall(msg.delta.new_element.exception, error)
         msg_str = msg.SerializeToString()
 
     return msg_str
 
 
-def _convert_msg_to_exception_msg(msg, e):
-    import streamlit.elements.exception_proto as exception_proto
-
-    delta_id = msg.metadata.delta_id
-    msg.Clear()
-    msg.metadata.delta_id = delta_id
-
-    exception_proto.marshall(msg.delta.new_element.exception, e)
-
-
 def is_url_from_allowed_origins(url):
     """Return True if URL is from allowed origins (for CORS purpose).
 


Test patch:
diff --git a/lib/tests/streamlit/Server_test.py b/lib/tests/streamlit/Server_test.py
--- a/lib/tests/streamlit/Server_test.py
+++ b/lib/tests/streamlit/Server_test.py
@@ -436,6 +436,22 @@ def test_should_cache_msg(self):
         config._set_option("global.minCachedMessageSize", 1000, "test")
         self.assertFalse(is_cacheable_msg(_create_dataframe_msg([1, 2, 3])))
 
+    def test_should_limit_msg_size(self):
+        # Set up a 60MB ForwardMsg string
+        large_msg = _create_dataframe_msg([1, 2, 3])
+        large_msg.delta.new_element.markdown.body = "X" * 60 * 1000 * 1000
+        # Create a copy, since serialize_forward_msg modifies the original proto
+        large_msg_copy = ForwardMsg()
+        large_msg_copy.CopyFrom(large_msg)
+        deserialized_msg = ForwardMsg()
+        deserialized_msg.ParseFromString(serialize_forward_msg(large_msg_copy))
+
+        # The metadata should be the same, but contents should be replaced
+        self.assertEqual(deserialized_msg.metadata, large_msg.metadata)
+        self.assertNotEqual(deserialized_msg, large_msg)
+        expected = "Data of size 60.0MB exceeds write limit of 50.0MB"
+        self.assertEqual(deserialized_msg.delta.new_element.exception.message, expected)
+
 
 class HealthHandlerTest(tornado.testing.AsyncHTTPTestCase):
     """Tests the /healthz endpoint"""