Repository: streamlit/streamlit

Issue:
Stale widgets fill screen after script reruns
### Checklist

- [X] I have searched the [existing issues](https://github.com/streamlit/streamlit/issues) for similar issues.
- [X] I added a very descriptive title to this issue.
- [X] I have provided sufficient information below to help reproduce this issue.

### Summary

If a script produces fewer elements than a previous time it was run, stale widgets begin to fill the screen and do not disappear. This is because the delta message to the frontend will add an element with the same widget ID as another element in the same `BlockNode`, causing a React "Encountered two children with the same key" error in the frontend.

### Reproducible Code Example

[![Open in Streamlit Cloud](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://issues.streamlitapp.com/?issue=gh-8360)

```Python
import streamlit as st
import time

if 'foo' not in st.session_state:
    st.write('Setting state')
    st.session_state['foo'] = 'bar'

if st.button('Run'):
    # Allow for forward message queue to flush button element
    time.sleep(1)

    del st.session_state['foo']
    st.rerun()
```


### Steps To Reproduce

1. Click on the "Run" button

### Expected Behavior

Only a single Run button is shown on the screen after clicking it.

### Current Behavior

A new stale Run button appears each time the button is clicked, filling the screen with buttons.

https://github.com/streamlit/streamlit/assets/5288285/73163485-69e6-496a-95ad-00b649753b18

```
react_devtools_backend_compact.js:13024 Warning: Encountered two children with the same key,
`$$WIDGET_ID-64a6367f129c80d7584b057250e50a31-None`. Keys should be unique so that
components maintain their identity across updates. Non-unique keys may cause children to be
duplicated and/or omitted — the behavior is unsupported and could change in a future version.
    at ChildRenderer (http://localhost:3000/static/js/bundle.js:15819:56)
    at div
    at http://localhost:3000/static/js/bundle.js:99124:66
    at div
    at http://localhost:3000/static/js/bundle.js:99124:66
    at div
    at http://localhost:3000/static/js/bundle.js:99124:66
    at VerticalBlock (http://localhost:3000/static/js/bundle.js:15906:71)
    at div
    at http://localhost:3000/static/js/bundle.js:99124:66
    at section
    at http://localhost:3000/static/js/bundle.js:99124:66
    at div
    at http://localhost:3000/static/js/bundle.js:99124:66
    at AppView (http://localhost:3000/static/js/bundle.js:2392:5)
    at div
    at http://localhost:3000/static/js/bundle.js:99124:66
    at div
    at FocusTrap (http://localhost:3000/static/js/bundle.js:209011:5)
    at HotKeys (http://localhost:3000/static/js/bundle.js:209259:5)
    at App (http://localhost:3000/static/js/bundle.js:293:5)
    at div
    at ComponentWithScreencast (http://localhost:3000/static/js/bundle.js:12790:7)
    at ThemeProvider (http://localhost:3000/static/js/bundle.js:99154:64)
    at ThemeProvider (http://localhost:3000/static/js/bundle.js:113703:21)
    at UIDReset (http://localhost:3000/static/js/bundle.js:218317:21)
    at div
    at http://localhost:3000/static/js/bundle.js:266981:58
    at Styled(div)
    at LayersManager (http://localhost:3000/static/js/bundle.js:104644:5)
    at BaseProvider (http://localhost:3000/static/js/bundle.js:102642:24)
    at RootStyleProvider (http://localhost:3000/static/js/bundle.js:14657:5)
    at ThemedApp (http://localhost:3000/static/js/bundle.js:2178:107)
    at DevProvider (http://localhost:3000/static/js/bundle.js:266709:5)
```

### Is this a regression?

- [ ] Yes, this used to work in a previous version.

### Debug info

- Streamlit version: 1.32.2
- Python version: 3.11.5
- Operating System: macOS 13.5.2
- Browser: Chrome


### Additional Information

The first time the script runs there are two deltas sent:
```
{"deltaPath":[0,0]},"delta":{"newElement":{"markdown":{"body":"Setting state","elementType":"NATIVE"}}}}
{"deltaPath":[0,1]},"delta":{"newElement":{"button":{"id":"$$WIDGET_ID-64a6367f129c80d7584b057250e50a31-None","label":"Run","type":"secondary"}}}}
```
so the main block node contains elements with keys:
```
[
    0,
    '$$WIDGET_ID-64a6367f129c80d7584b057250e50a31-None',
]
```

After clicking the Run button, only one delta is sent:
```
{"deltaPath":[0,0]},"delta":{"newElement":{"button":{"id":"$$WIDGET_ID-64a6367f129c80d7584b057250e50a31-None","label":"Run","type":"secondary"}}}}
```
so the main block node temporarily contains elements with keys
```
[
    '$$WIDGET_ID-64a6367f129c80d7584b057250e50a31-None',
    '$$WIDGET_ID-64a6367f129c80d7584b057250e50a31-None',
]
```
which is not allowed, triggering "Encountered two children with the same key" error and causing the stale widget to remain on the screen.

See:
https://github.com/streamlit/streamlit/blob/4dbc82b90b51173d292bcbc7f93df38da39ac8ec/frontend/lib/src/components/core/Block/Block.tsx#L201-L202

Removing the use of the widget ID seems to fix the issue but is slower:
```diff
- const key = getElementWidgetID(node.element) || index 
+ const key = index 
```

Possibly related issue: https://github.com/streamlit/streamlit/issues/2820




Discussion:
**If this issue affects you, please react with a 👍 (thumbs up emoji) to the initial post.**

Your feedback helps us prioritize which bugs to investigate and address first.

![Visits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fstreamlit%2Fstreamlit%2Fissues%2F8360&title=visits&edge_flat=false)

Thanks for bringing this up.

Looking into this more, it looks like this issue has been here since forever from `st.experimental_rerun`.
I can also reproduce that the proposed fix does seemingly fix the issue. However, I would have to think about the implications of this.
Thanks for reviewing, it is also possible to trigger the "Encountered two children with the same key" error without `st.rerun` in the code example, so I think there could be cases that cause this issue without using `st.rerun`. However without `st.rerun`, it does not seem to fill the screen with stale widgets, maybe by chance.


Resolving patch:
diff --git a/lib/streamlit/runtime/app_session.py b/lib/streamlit/runtime/app_session.py
--- a/lib/streamlit/runtime/app_session.py
+++ b/lib/streamlit/runtime/app_session.py
@@ -465,7 +465,7 @@ def _on_pages_changed(self, _) -> None:
             self._local_sources_watcher.update_watched_pages()
 
     def _clear_queue(self) -> None:
-        self._browser_queue.clear()
+        self._browser_queue.clear(retain_lifecycle_msgs=True)
 
     def _on_scriptrunner_event(
         self,
diff --git a/lib/streamlit/runtime/forward_msg_queue.py b/lib/streamlit/runtime/forward_msg_queue.py
--- a/lib/streamlit/runtime/forward_msg_queue.py
+++ b/lib/streamlit/runtime/forward_msg_queue.py
@@ -79,9 +79,29 @@ def enqueue(self, msg: ForwardMsg) -> None:
         self._delta_index_map[delta_key] = len(self._queue)
         self._queue.append(msg)
 
-    def clear(self) -> None:
-        """Clear the queue."""
-        self._queue = []
+    def clear(self, retain_lifecycle_msgs: bool = False) -> None:
+        """Clear the queue, potentially retaining lifecycle messages.
+
+        The retain_lifecycle_msgs argument exists because in some cases (in particular
+        when a currently running script is interrupted by a new BackMsg), we don't want
+        to remove certain messages from the queue as doing so may cause the client to
+        not hear about important script lifecycle events (such as the script being
+        stopped early in order to be rerun).
+        """
+        if not retain_lifecycle_msgs:
+            self._queue = []
+        else:
+            self._queue = [
+                msg
+                for msg in self._queue
+                if msg.WhichOneof("type")
+                in {
+                    "script_finished",
+                    "session_status_changed",
+                    "parent_message",
+                }
+            ]
+
         self._delta_index_map = dict()
 
     def flush(self) -> list[ForwardMsg]:


Test patch:
diff --git a/lib/tests/streamlit/runtime/app_session_test.py b/lib/tests/streamlit/runtime/app_session_test.py
--- a/lib/tests/streamlit/runtime/app_session_test.py
+++ b/lib/tests/streamlit/runtime/app_session_test.py
@@ -831,7 +831,9 @@ async def test_handle_backmsg_exception(self):
             side_effect=lambda msg: forward_msg_queue_events.append(msg)
         )
         mock_queue.clear = MagicMock(
-            side_effect=lambda: forward_msg_queue_events.append(CLEAR_QUEUE)
+            side_effect=lambda retain_lifecycle_msgs: forward_msg_queue_events.append(
+                CLEAR_QUEUE
+            )
         )
 
         session._browser_queue = mock_queue
diff --git a/lib/tests/streamlit/runtime/forward_msg_queue_test.py b/lib/tests/streamlit/runtime/forward_msg_queue_test.py
--- a/lib/tests/streamlit/runtime/forward_msg_queue_test.py
+++ b/lib/tests/streamlit/runtime/forward_msg_queue_test.py
@@ -60,30 +60,30 @@
 class ForwardMsgQueueTest(unittest.TestCase):
     def test_simple_enqueue(self):
         """Enqueue a single ForwardMsg."""
-        rq = ForwardMsgQueue()
-        self.assertTrue(rq.is_empty())
+        fmq = ForwardMsgQueue()
+        self.assertTrue(fmq.is_empty())
 
-        rq.enqueue(NEW_SESSION_MSG)
+        fmq.enqueue(NEW_SESSION_MSG)
 
-        self.assertFalse(rq.is_empty())
-        queue = rq.flush()
-        self.assertTrue(rq.is_empty())
+        self.assertFalse(fmq.is_empty())
+        queue = fmq.flush()
+        self.assertTrue(fmq.is_empty())
         self.assertEqual(1, len(queue))
         self.assertTrue(queue[0].new_session.config.allow_run_on_save)
 
     def test_enqueue_two(self):
         """Enqueue two ForwardMsgs."""
-        rq = ForwardMsgQueue()
-        self.assertTrue(rq.is_empty())
+        fmq = ForwardMsgQueue()
+        self.assertTrue(fmq.is_empty())
 
-        rq.enqueue(NEW_SESSION_MSG)
+        fmq.enqueue(NEW_SESSION_MSG)
 
         TEXT_DELTA_MSG1.metadata.delta_path[:] = make_delta_path(
             RootContainer.MAIN, (), 0
         )
-        rq.enqueue(TEXT_DELTA_MSG1)
+        fmq.enqueue(TEXT_DELTA_MSG1)
 
-        queue = rq.flush()
+        queue = fmq.flush()
         self.assertEqual(2, len(queue))
         self.assertEqual(
             make_delta_path(RootContainer.MAIN, (), 0), queue[1].metadata.delta_path
@@ -92,22 +92,22 @@ def test_enqueue_two(self):
 
     def test_enqueue_three(self):
         """Enqueue 3 ForwardMsgs."""
-        rq = ForwardMsgQueue()
-        self.assertTrue(rq.is_empty())
+        fmq = ForwardMsgQueue()
+        self.assertTrue(fmq.is_empty())
 
-        rq.enqueue(NEW_SESSION_MSG)
+        fmq.enqueue(NEW_SESSION_MSG)
 
         TEXT_DELTA_MSG1.metadata.delta_path[:] = make_delta_path(
             RootContainer.MAIN, (), 0
         )
-        rq.enqueue(TEXT_DELTA_MSG1)
+        fmq.enqueue(TEXT_DELTA_MSG1)
 
         TEXT_DELTA_MSG2.metadata.delta_path[:] = make_delta_path(
             RootContainer.MAIN, (), 1
         )
-        rq.enqueue(TEXT_DELTA_MSG2)
+        fmq.enqueue(TEXT_DELTA_MSG2)
 
-        queue = rq.flush()
+        queue = fmq.flush()
         self.assertEqual(3, len(queue))
         self.assertEqual(
             make_delta_path(RootContainer.MAIN, (), 0), queue[1].metadata.delta_path
@@ -122,22 +122,22 @@ def test_replace_element(self):
         """Enqueuing an element with the same delta_path as another element
         already in the queue should replace the original element.
         """
-        rq = ForwardMsgQueue()
-        self.assertTrue(rq.is_empty())
+        fmq = ForwardMsgQueue()
+        self.assertTrue(fmq.is_empty())
 
-        rq.enqueue(NEW_SESSION_MSG)
+        fmq.enqueue(NEW_SESSION_MSG)
 
         TEXT_DELTA_MSG1.metadata.delta_path[:] = make_delta_path(
             RootContainer.MAIN, (), 0
         )
-        rq.enqueue(TEXT_DELTA_MSG1)
+        fmq.enqueue(TEXT_DELTA_MSG1)
 
         TEXT_DELTA_MSG2.metadata.delta_path[:] = make_delta_path(
             RootContainer.MAIN, (), 0
         )
-        rq.enqueue(TEXT_DELTA_MSG2)
+        fmq.enqueue(TEXT_DELTA_MSG2)
 
-        queue = rq.flush()
+        queue = fmq.flush()
         self.assertEqual(2, len(queue))
         self.assertEqual(
             make_delta_path(RootContainer.MAIN, (), 0), queue[1].metadata.delta_path
@@ -148,8 +148,8 @@ def test_replace_element(self):
     def test_dont_replace_block(self, other_msg: ForwardMsg):
         """add_block deltas should never be replaced because they can
         have dependent deltas later in the queue."""
-        rq = ForwardMsgQueue()
-        self.assertTrue(rq.is_empty())
+        fmq = ForwardMsgQueue()
+        self.assertTrue(fmq.is_empty())
 
         ADD_BLOCK_MSG.metadata.delta_path[:] = make_delta_path(
             RootContainer.MAIN, (), 0
@@ -159,34 +159,34 @@ def test_dont_replace_block(self, other_msg: ForwardMsg):
 
         # Delta messages should not replace `add_block` deltas with the
         # same delta_path.
-        rq.enqueue(ADD_BLOCK_MSG)
-        rq.enqueue(other_msg)
-        queue = rq.flush()
+        fmq.enqueue(ADD_BLOCK_MSG)
+        fmq.enqueue(other_msg)
+        queue = fmq.flush()
         self.assertEqual(2, len(queue))
         self.assertEqual(ADD_BLOCK_MSG, queue[0])
         self.assertEqual(other_msg, queue[1])
 
     def test_multiple_containers(self):
         """Deltas should only be coalesced if they're in the same container"""
-        rq = ForwardMsgQueue()
-        self.assertTrue(rq.is_empty())
+        fmq = ForwardMsgQueue()
+        self.assertTrue(fmq.is_empty())
 
-        rq.enqueue(NEW_SESSION_MSG)
+        fmq.enqueue(NEW_SESSION_MSG)
 
         def enqueue_deltas(container: int, path: Tuple[int, ...]):
             # We deep-copy the protos because we mutate each one
             # multiple times.
             msg = copy.deepcopy(TEXT_DELTA_MSG1)
             msg.metadata.delta_path[:] = make_delta_path(container, path, 0)
-            rq.enqueue(msg)
+            fmq.enqueue(msg)
 
             msg = copy.deepcopy(DF_DELTA_MSG)
             msg.metadata.delta_path[:] = make_delta_path(container, path, 1)
-            rq.enqueue(msg)
+            fmq.enqueue(msg)
 
             msg = copy.deepcopy(ADD_ROWS_MSG)
             msg.metadata.delta_path[:] = make_delta_path(container, path, 1)
-            rq.enqueue(msg)
+            fmq.enqueue(msg)
 
         enqueue_deltas(RootContainer.MAIN, ())
         enqueue_deltas(RootContainer.SIDEBAR, (0, 0, 1))
@@ -198,8 +198,38 @@ def assert_deltas(container: int, path: Tuple[int, ...], idx: int):
             )
             self.assertEqual("text1", queue[idx].delta.new_element.text.body)
 
-        queue = rq.flush()
+        queue = fmq.flush()
         self.assertEqual(7, len(queue))
 
         assert_deltas(RootContainer.MAIN, (), 1)
         assert_deltas(RootContainer.SIDEBAR, (0, 0, 1), 4)
+
+    def test_clear_retain_lifecycle_msgs(self):
+        fmq = ForwardMsgQueue()
+
+        script_finished_msg = ForwardMsg()
+        script_finished_msg.script_finished = (
+            ForwardMsg.ScriptFinishedStatus.FINISHED_EARLY_FOR_RERUN
+        )
+
+        session_status_changed_msg = ForwardMsg()
+        session_status_changed_msg.session_status_changed.script_is_running = True
+
+        parent_msg = ForwardMsg()
+        parent_msg.parent_message.message = "hello"
+
+        fmq.enqueue(NEW_SESSION_MSG)
+        fmq.enqueue(TEXT_DELTA_MSG1)
+        fmq.enqueue(script_finished_msg)
+        fmq.enqueue(session_status_changed_msg)
+        fmq.enqueue(parent_msg)
+
+        fmq.clear(retain_lifecycle_msgs=True)
+        assert fmq._queue == [
+            script_finished_msg,
+            session_status_changed_msg,
+            parent_msg,
+        ]
+
+        fmq.clear()
+        assert fmq._queue == []