Repository: streamlit/streamlit
Issue category: Performance Issue

Issue:
Streamlit unnecessarily runs the file watcher against every path in imported modules, even when watching is disabled
### Checklist

- [X] I have searched the [existing issues](https://github.com/streamlit/streamlit/issues) for similar issues.
- [X] I added a very descriptive title to this issue.
- [X] I have provided sufficient information below to help reproduce this issue.

### Summary

We are running an application that has a bunch of module dependencies.
Recording some profiles, we often see stack traces like this

![image](https://github.com/streamlit/streamlit/assets/84412986/108c2645-f57f-4d0c-ac48-8c5e0729a6eb)

https://github.com/streamlit/streamlit/blob/ad3dc4e19d261fc66560a25bab92a7a560068338/lib/streamlit/runtime/app_session.py#L604

This happens despite the file watcher type being set to `none`, because the local watcher does not gate on that setting. It is only when the per-file watch request addition happens that the setting is checked and it determines that no watch is required. This is incredibly wasteful as it incurs a `isfile` check (along with an fnmatch) for ~40000 files in our case. If I understand the code correctly, this happens at the end of every successful page load.

This seems like it could easily be avoided if the app session also consulted the setting.

Does this make sense and would you accept the relevant fix?

### Reproducible Code Example

_No response_

### Steps To Reproduce

_No response_

### Expected Behavior

Don't iterate over a bunch of files when file watching is disabled.

### Current Behavior

Lot of compute and disk I/O wasted on "watching" files.

### Is this a regression?

- [ ] Yes, this used to work in a previous version.

### Debug info

- Streamlit version: We are running 1.33, but the same code exists in the latest code
- Python version: 3.8
- Operating System: Linux
- Browser: Any


### Additional Information

_No response_


Hints:
**If this issue affects you, please react with a 👍 (thumbs up emoji) to the initial post.**

Your feedback helps us prioritize which bugs to investigate and address first.

![Visits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fstreamlit%2Fstreamlit%2Fissues%2F8738&title=visits&edge_flat=false)

Hi @nikhil-marathe-skydio,

Whoops, this definitely seems like an oversight on our end. I think the ideal thing to do here would be to avoid instantiating a `LocalSourcesWatcher` entirely if file watching is disabled. I'll go ahead and create a PR to fix this myself given it should just be a one line change.

Patch:
diff --git a/lib/streamlit/runtime/app_session.py b/lib/streamlit/runtime/app_session.py
index dc9a13c17511..f68b216929db 100644
--- a/lib/streamlit/runtime/app_session.py
+++ b/lib/streamlit/runtime/app_session.py
@@ -144,7 +144,8 @@ def __init__(
         self._stop_config_listener: Callable[[], bool] | None = None
         self._stop_pages_listener: Callable[[], None] | None = None
 
-        self.register_file_watchers()
+        if config.get_option("server.fileWatcherType") != "none":
+            self.register_file_watchers()
 
         self._run_on_save = config.get_option("server.runOnSave")
 


Test patch:
diff --git a/lib/tests/streamlit/runtime/app_session_test.py b/lib/tests/streamlit/runtime/app_session_test.py
index ba999fc4d967..a94a00d68468 100644
--- a/lib/tests/streamlit/runtime/app_session_test.py
+++ b/lib/tests/streamlit/runtime/app_session_test.py
@@ -125,13 +125,13 @@ def test_shutdown(self, patched_disconnect):
         session._uploaded_file_mgr = mock_file_mgr
 
         session.shutdown()
-        self.assertEqual(AppSessionState.SHUTDOWN_REQUESTED, session._state)
+        assert AppSessionState.SHUTDOWN_REQUESTED == session._state
         mock_file_mgr.remove_session_files.assert_called_once_with(session.id)
         patched_disconnect.assert_called_once_with(session._on_secrets_file_changed)
 
         # A 2nd shutdown call should have no effect.
         session.shutdown()
-        self.assertEqual(AppSessionState.SHUTDOWN_REQUESTED, session._state)
+        assert AppSessionState.SHUTDOWN_REQUESTED == session._state
 
         mock_file_mgr.remove_session_files.assert_called_once_with(session.id)
 
@@ -173,11 +173,11 @@ def test_unique_id(self):
         """Each AppSession should have a unique ID"""
         session1 = _create_test_session()
         session2 = _create_test_session()
-        self.assertNotEqual(session1.id, session2.id)
+        assert session1.id != session2.id
 
     def test_creates_session_state_on_init(self):
         session = _create_test_session()
-        self.assertIsInstance(session.session_state, SessionState)
+        assert isinstance(session.session_state, SessionState)
 
     def test_creates_fragment_storage_on_init(self):
         session = _create_test_session()
@@ -185,13 +185,13 @@ def test_creates_fragment_storage_on_init(self):
         # isinstance (there's no need to as the static type checker already ensures
         # the field has the correct type), and we don't want to mark
         # MemoryFragmentStorage as @runtime_checkable.
-        self.assertIsNotNone(session._fragment_storage)
+        assert session._fragment_storage is not None
 
     def test_clear_cache_resets_session_state(self):
         session = _create_test_session()
         session._session_state["foo"] = "bar"
         session._handle_clear_cache_request()
-        self.assertTrue("foo" not in session._session_state)
+        assert "foo" not in session._session_state
 
     @patch("streamlit.runtime.legacy_caching.clear_cache")
     @patch("streamlit.runtime.caching.cache_data.clear")
@@ -302,7 +302,7 @@ def test_rerun_fragment_requests_existing_scriptrunner(
     def test_create_scriptrunner(self, mock_scriptrunner: MagicMock):
         """Test that _create_scriptrunner does what it should."""
         session = _create_test_session()
-        self.assertIsNone(session._scriptrunner)
+        assert session._scriptrunner is None
 
         session._create_scriptrunner(initial_rerun_data=RerunData())
 
@@ -318,7 +318,7 @@ def test_create_scriptrunner(self, mock_scriptrunner: MagicMock):
             fragment_storage=session._fragment_storage,
         )
 
-        self.assertIsNotNone(session._scriptrunner)
+        assert session._scriptrunner is not None
 
         # And that the ScriptRunner was initialized and started.
         scriptrunner: MagicMock = cast(MagicMock, session._scriptrunner)
@@ -376,7 +376,7 @@ def test_resets_debug_last_backmsg_id_on_script_finished(self):
                 forward_msg=ForwardMsg(),
             )
 
-            self.assertIsNone(session._debug_last_backmsg_id)
+            assert session._debug_last_backmsg_id is None
 
     @patch("streamlit.runtime.app_session.ScriptRunner", MagicMock(spec=ScriptRunner))
     @patch("streamlit.runtime.app_session.AppSession._enqueue_forward_msg", MagicMock())
@@ -395,7 +395,7 @@ def test_sets_state_to_not_running_on_rerun_event(self):
                 forward_msg=ForwardMsg(),
             )
 
-            self.assertEqual(session._state, AppSessionState.APP_NOT_RUNNING)
+            assert session._state == AppSessionState.APP_NOT_RUNNING
 
     def test_passes_client_state_on_run_on_save(self):
         session = _create_test_session()
@@ -419,7 +419,7 @@ def test_does_not_rerun_if_not_current_page(self):
         # Clearing the cache should still have been called
         session._script_cache.clear.assert_called_once()
 
-        self.assertEqual(session.request_rerun.called, False)
+        assert not session.request_rerun.called
 
     @patch(
         "streamlit.runtime.app_session.source_util.get_pages",
@@ -468,7 +468,7 @@ def test_tags_fwd_msgs_with_last_backmsg_id_if_set(self):
         msg = ForwardMsg()
         session._enqueue_forward_msg(msg)
 
-        self.assertEqual(msg.debug_last_backmsg_id, "some backmsg id")
+        assert msg.debug_last_backmsg_id == "some backmsg id"
 
     @patch("streamlit.runtime.app_session.config.on_config_parsed")
     @patch("streamlit.runtime.app_session.source_util.register_pages_changed_callback")
@@ -501,7 +501,12 @@ def test_recreates_local_sources_watcher_if_none(self):
         session._local_sources_watcher = None
 
         session.register_file_watchers()
-        self.assertIsNotNone(session._local_sources_watcher)
+        assert session._local_sources_watcher
+
+    @patch_config_options({"server.fileWatcherType": "none"})
+    def test_no_local_sources_watcher_if_file_watching_disabled(self):
+        session = _create_test_session()
+        assert not session._local_sources_watcher
 
     @patch(
         "streamlit.runtime.app_session.secrets_singleton.file_change_listener.disconnect"
@@ -525,9 +530,9 @@ def test_disconnect_file_watchers(self, patched_secrets_disconnect):
                 session._on_secrets_file_changed
             )
 
-            self.assertIsNone(session._local_sources_watcher)
-            self.assertIsNone(session._stop_config_listener)
-            self.assertIsNone(session._stop_pages_listener)
+            assert session._local_sources_watcher is None
+            assert session._stop_config_listener is None
+            assert session._stop_pages_listener is None
 
     def test_disconnect_file_watchers_removes_refs(self):
         """Test that calling disconnect_file_watchers on the AppSession
@@ -538,7 +543,7 @@ def test_disconnect_file_watchers_removes_refs(self):
 
         # Various listeners should have references to session file/pages/secrets changed
         # handlers.
-        self.assertGreater(len(gc.get_referrers(session)), 0)
+        assert len(gc.get_referrers(session)) > 0
 
         session.disconnect_file_watchers()
 
@@ -552,7 +557,7 @@ def test_disconnect_file_watchers_removes_refs(self):
         gc.collect(2)
         gc.collect(2)
 
-        self.assertEqual(len(gc.get_referrers(session)), 0)
+        assert len(gc.get_referrers(session)) == 0
 
     @patch("streamlit.runtime.app_session.AppSession._enqueue_forward_msg")
     def test_handle_file_urls_request(self, mock_enqueue):
@@ -683,34 +688,31 @@ async def test_enqueue_new_session_message(self):
         await asyncio.sleep(0)
 
         sent_messages = session._browser_queue._queue
-        self.assertEqual(2, len(sent_messages))  # NewApp and SessionState messages
+        assert len(sent_messages) == 2  # NewApp and SessionState messages
         session._clear_queue.assert_called_once()
 
         # Note that we're purposefully not very thoroughly testing new_session
         # fields below to avoid getting to the point where we're just
         # duplicating code in tests.
         new_session_msg = sent_messages[0].new_session
-        self.assertEqual("mock_scriptrun_id", new_session_msg.script_run_id)
+        assert new_session_msg.script_run_id == "mock_scriptrun_id"
 
-        self.assertTrue(new_session_msg.HasField("config"))
-        self.assertEqual(
-            config.get_option("server.allowRunOnSave"),
-            new_session_msg.config.allow_run_on_save,
+        assert new_session_msg.HasField("config")
+        assert (
+            config.get_option("server.allowRunOnSave")
+            == new_session_msg.config.allow_run_on_save
         )
 
-        self.assertTrue(new_session_msg.HasField("custom_theme"))
-        self.assertEqual("black", new_session_msg.custom_theme.text_color)
+        assert new_session_msg.HasField("custom_theme")
+        assert new_session_msg.custom_theme.text_color == "black"
 
         init_msg = new_session_msg.initialize
-        self.assertTrue(init_msg.HasField("user_info"))
+        assert init_msg.HasField("user_info")
 
-        self.assertEqual(
-            list(new_session_msg.app_pages),
-            [
-                AppPage(page_script_hash="hash1", page_name="page1", icon=""),
-                AppPage(page_script_hash="hash2", page_name="page2", icon="🎉"),
-            ],
-        )
+        assert list(new_session_msg.app_pages) == [
+            AppPage(page_script_hash="hash1", page_name="page1", icon=""),
+            AppPage(page_script_hash="hash2", page_name="page2", icon="🎉"),
+        ]
 
         add_script_run_ctx(ctx=orig_ctx)
 
@@ -751,11 +753,11 @@ async def test_new_session_message_includes_fragment_ids(self):
         await asyncio.sleep(0)
 
         sent_messages = session._browser_queue._queue
-        self.assertEqual(2, len(sent_messages))  # NewApp and SessionState messages
+        assert len(sent_messages) == 2  # NewApp and SessionState messages
         session._clear_queue.assert_not_called()
 
         new_session_msg = sent_messages[0].new_session
-        self.assertEqual(new_session_msg.fragment_ids_this_run, ["my_fragment_id"])
+        assert new_session_msg.fragment_ids_this_run == ["my_fragment_id"]
 
         add_script_run_ctx(ctx=orig_ctx)
 
@@ -799,7 +801,7 @@ async def test_event_handler_asserts_if_called_off_event_loop(self):
             "streamlit.runtime.app_session.asyncio.get_running_loop",
             return_value=MagicMock(),
         ):
-            with self.assertRaises(AssertionError):
+            with pytest.raises(AssertionError):
                 session._handle_scriptrunner_event_on_event_loop(
                     sender=MagicMock(), event=ScriptRunnerEvent.SCRIPT_STARTED
                 )
@@ -842,7 +844,7 @@ async def test_handle_backmsg_exception(self):
 
         # Messages get sent in an eventloop callback, which hasn't had a chance
         # to run yet. Our message queue should be empty.
-        self.assertEqual([], forward_msg_queue_events)
+        assert forward_msg_queue_events == []
 
         # Run callbacks
         await asyncio.sleep(0)
@@ -874,8 +876,7 @@ async def test_handle_backmsg_exception(self):
                 ]
             )
 
-        # Assert the results!
-        self.assertEqual(expected_events, forward_msg_queue_events)
+        assert expected_events == forward_msg_queue_events
 
     async def test_handle_backmsg_handles_exceptions(self):
         """Exceptions raised in handle_backmsg should be sent to
@@ -904,7 +905,7 @@ async def test_handle_backmsg_handles_debug_ids(self):
             rerun_script=session._client_state, debug_last_backmsg_id="some backmsg"
         )
         session.handle_backmsg(msg)
-        self.assertEqual(session._debug_last_backmsg_id, "some backmsg")
+        assert session._debug_last_backmsg_id == "some backmsg"
 
     @patch("streamlit.runtime.app_session._LOGGER")
     async def test_handles_app_heartbeat_backmsg(self, patched_logger):
@@ -943,7 +944,7 @@ def test_no_custom_theme_prop_if_no_theme(self, patched_config):
         new_session_msg = msg.new_session
         app_session._populate_theme_msg(new_session_msg.custom_theme)
 
-        self.assertEqual(new_session_msg.HasField("custom_theme"), False)
+        assert not new_session_msg.HasField("custom_theme")
 
     @patch("streamlit.runtime.app_session.config")
     def test_can_specify_some_options(self, patched_config):
@@ -960,11 +961,11 @@ def test_can_specify_some_options(self, patched_config):
         new_session_msg = msg.new_session
         app_session._populate_theme_msg(new_session_msg.custom_theme)
 
-        self.assertEqual(new_session_msg.HasField("custom_theme"), True)
-        self.assertEqual(new_session_msg.custom_theme.primary_color, "coral")
+        assert new_session_msg.HasField("custom_theme")
+        assert new_session_msg.custom_theme.primary_color == "coral"
         # In proto3, primitive fields are technically always required and are
         # set to the type's zero value when undefined.
-        self.assertEqual(new_session_msg.custom_theme.background_color, "")
+        assert new_session_msg.custom_theme.background_color == ""
 
     @patch("streamlit.runtime.app_session.config")
     def test_can_specify_all_options(self, patched_config):
@@ -977,9 +978,9 @@ def test_can_specify_all_options(self, patched_config):
         new_session_msg = msg.new_session
         app_session._populate_theme_msg(new_session_msg.custom_theme)
 
-        self.assertEqual(new_session_msg.HasField("custom_theme"), True)
-        self.assertEqual(new_session_msg.custom_theme.primary_color, "coral")
-        self.assertEqual(new_session_msg.custom_theme.background_color, "white")
+        assert new_session_msg.HasField("custom_theme")
+        assert new_session_msg.custom_theme.primary_color == "coral"
+        assert new_session_msg.custom_theme.background_color == "white"
 
     @patch("streamlit.runtime.app_session._LOGGER")
     @patch("streamlit.runtime.app_session.config")
@@ -1028,18 +1029,16 @@ def test_returns_true_if_current_page_changed(self):
         session = _create_test_session()
         session._client_state.page_script_hash = "hash2"
 
-        self.assertEqual(session._should_rerun_on_file_change("page2.py"), True)
+        assert session._should_rerun_on_file_change("page2.py")
 
     def test_returns_true_if_changed_file_is_not_page(self):
         session = _create_test_session()
         session._client_state.page_script_hash = "hash1"
 
-        self.assertEqual(
-            session._should_rerun_on_file_change("some_other_file.py"), True
-        )
+        assert session._should_rerun_on_file_change("some_other_file.py")
 
     def test_returns_false_if_different_page_changed(self):
         session = _create_test_session()
         session._client_state.page_script_hash = "hash2"
 
-        self.assertEqual(session._should_rerun_on_file_change("page1.py"), False)
+        assert not session._should_rerun_on_file_change("page1.py")