Repository: AgnostiqHQ/covalent
Pull request #1607 (2023-04-28 12:08:50)

defer creation of server specific config entries
Fixes https://github.com/AgnostiqHQ/covalent/issues/1430

- [x] I have added the tests to cover my changes.
- [x] I have updated the documentation and CHANGELOG accordingly.
- [x] I have read the CONTRIBUTING document.


codecov[bot]: ## [Codecov](https://codecov.io/gh/AgnostiqHQ/covalent/pull/1607?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AgnostiqHQ) Report
> Merging [#1607](https://codecov.io/gh/AgnostiqHQ/covalent/pull/1607?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AgnostiqHQ) (fa68cba) into [develop](https://codecov.io/gh/AgnostiqHQ/covalent/commit/5172f98c8b967f15364a2f6c35d47294eff4ba7b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AgnostiqHQ) (5172f98) will **increase** coverage by `1.31%`.
> The diff coverage is `100.00%`.

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://codecov.io/gh/AgnostiqHQ/covalent/pull/1607/graphs/tree.svg?width=650&height=150&src=pr&token=YGHCB3DE4P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AgnostiqHQ)](https://codecov.io/gh/AgnostiqHQ/covalent/pull/1607?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=AgnostiqHQ)

```diff
@@             Coverage Diff             @@
##           develop    #1607      +/-   ##
===========================================
+ Coverage    85.94%   87.26%   +1.31%     
===========================================
  Files          159       83      -76     
  Lines         6658     4969    -1689     
  Branches       145        0     -145     
===========================================
- Hits          5722     4336    -1386     
+ Misses         834      633     -201     
+ Partials       102        0     -102     
```

| Flag | Coverage Δ | |
|---|---|---|
| Dispatcher | `87.26% <100.00%> (-0.02%)` | :arrow_down: |
| Functional_Tests | `?` | |
| SDK | `87.25% <ø> (-0.06%)` | :arrow_down: |
| UI_Frontend | `?` | |

</details>
madhur-tandon: Could appreciate a review @cjao 
CC: @AlejandroEsquivel / @kessler-frost 
madhur-tandon: @kessler-frost can you re-trigger the RTD build...
santoshkumarradha: Thanks again @madhur-tandon. While @kessler-frost  does that, seems like RTD is failing with this error (I assume retreggiring might not help )

<details open>
<summary> Error message </summary>
<br>
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/util/inspect.py", line 412, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: module 'covalent.triggers' has no attribute 'dispatch'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/ext/autodoc/importer.py", line 110, in import_object
    obj = attrgetter(obj, mangled_name)
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 318, in get_attr
    return autodoc_attrgetter(self.env.app, obj, name, *defargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 2606, in autodoc_attrgetter
    return safe_getattr(obj, name, *defargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/util/inspect.py", line 428, in safe_getattr
    raise AttributeError(name) from exc
AttributeError: dispatch

WARNING: autodoc: failed to import function 'dispatch_sync' from module 'covalent.triggers'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/util/inspect.py", line 412, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: module 'covalent.triggers' has no attribute 'dispatch_sync'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/ext/autodoc/importer.py", line 110, in import_object
    obj = attrgetter(obj, mangled_name)
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 318, in get_attr
    return autodoc_attrgetter(self.env.app, obj, name, *defargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 2606, in autodoc_attrgetter
    return safe_getattr(obj, name, *defargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/covalent/envs/1607/lib/python3.8/site-packages/sphinx/util/inspect.py", line 428, in safe_getattr
    raise AttributeError(name) from exc
AttributeError: dispatch_sync
</details>


madhur-tandon: @santoshkumarradha I think re-trigger might help since my PR has nothing to do with the error encountered -- so maybe it's flaky at best...

But I can try investigating if re-trigger fails as well...
I say because before my last commit, everything passed...
madhur-tandon: @santoshkumarradha can you re-trigger the tests? docs passed but tests are flaky this time...
madhur-tandon: @santoshkumarradha This should be ready as well...

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8e551db4..54cfd664 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Raise error on dispatching a non-lattice
 - Helpful message when trying to dispatch a workflow when covalent server is not available
 - Open UI preview of transport graph when `lattice.draw()` is invoked and print URL of the same
+- Defer creation of server specific config entries until covalent is started
 
 ### Authors
 
diff --git a/covalent/_shared_files/config.py b/covalent/_shared_files/config.py
index 1c8ebadc..e24405a8 100644
--- a/covalent/_shared_files/config.py
+++ b/covalent/_shared_files/config.py
@@ -59,11 +59,6 @@ class ConfigManager:
 
         Path(self.get("sdk.log_dir")).mkdir(parents=True, exist_ok=True)
         Path(self.get("sdk.executor_dir")).mkdir(parents=True, exist_ok=True)
-        Path(self.get("dispatcher.cache_dir")).mkdir(parents=True, exist_ok=True)
-        Path(self.get("dispatcher.results_dir")).mkdir(parents=True, exist_ok=True)
-        Path(self.get("dispatcher.log_dir")).mkdir(parents=True, exist_ok=True)
-        Path(self.get("user_interface.log_dir")).mkdir(parents=True, exist_ok=True)
-        Path(self.get("dispatcher.db_path")).parent.mkdir(parents=True, exist_ok=True)
 
     def generate_default_config(self) -> None:
         """
diff --git a/covalent/_shared_files/utils.py b/covalent/_shared_files/utils.py
index 9c773265..dbbb3135 100644
--- a/covalent/_shared_files/utils.py
+++ b/covalent/_shared_files/utils.py
@@ -26,10 +26,19 @@ from datetime import timedelta
 from typing import Callable, Dict, Set, Tuple
 
 from . import logger
+from .config import get_config
 
 app_log = logger.app_log
 log_stack_info = logger.log_stack_info
 
+DEFAULT_UI_ADDRESS = get_config("user_interface.address")
+DEFAULT_UI_PORT = get_config("user_interface.port")
+
+
+def get_ui_url(path):
+    baseUrl = f"http://{DEFAULT_UI_ADDRESS}:{DEFAULT_UI_PORT}"
+    return f"{baseUrl}{path}"
+
 
 def get_random_available_port() -> int:
     """
diff --git a/covalent/_workflow/lattice.py b/covalent/_workflow/lattice.py
index 0da77e57..a4fb5756 100644
--- a/covalent/_workflow/lattice.py
+++ b/covalent/_workflow/lattice.py
@@ -35,7 +35,7 @@ from .._shared_files import logger
 from .._shared_files.config import get_config
 from .._shared_files.context_managers import active_lattice_manager
 from .._shared_files.defaults import DefaultMetadataValues
-from .._shared_files.utils import get_named_params, get_serialized_function_str
+from .._shared_files.utils import get_named_params, get_serialized_function_str, get_ui_url
 from .depsbash import DepsBash
 from .depscall import DepsCall
 from .depspip import DepsPip
@@ -268,7 +268,6 @@ class Lattice:
         """
 
         import covalent_ui.result_webhook as result_webhook
-        from covalent_ui.result_webhook import get_ui_url
 
         self.build_graph(*args, **kwargs)
         result_webhook.send_draw_request(self)
diff --git a/covalent_dispatcher/_cli/service.py b/covalent_dispatcher/_cli/service.py
index ddb33c68..41f4cc59 100644
--- a/covalent_dispatcher/_cli/service.py
+++ b/covalent_dispatcher/_cli/service.py
@@ -28,6 +28,7 @@ import os
 import shutil
 import socket
 import time
+from pathlib import Path
 from subprocess import DEVNULL, Popen
 from typing import Optional
 
@@ -219,6 +220,13 @@ def _graceful_start(
         click.echo(f"Covalent Triggers server has started at {dispatcher_addr}")
     else:
         click.echo(f"Covalent server has started at {dispatcher_addr}")
+
+    Path(get_config("dispatcher.cache_dir")).mkdir(parents=True, exist_ok=True)
+    Path(get_config("dispatcher.results_dir")).mkdir(parents=True, exist_ok=True)
+    Path(get_config("dispatcher.log_dir")).mkdir(parents=True, exist_ok=True)
+    Path(get_config("user_interface.log_dir")).mkdir(parents=True, exist_ok=True)
+    Path(get_config("dispatcher.db_path")).parent.mkdir(parents=True, exist_ok=True)
+
     return port
 
 
diff --git a/covalent_ui/result_webhook.py b/covalent_ui/result_webhook.py
index 94e9aabb..609e2da4 100644
--- a/covalent_ui/result_webhook.py
+++ b/covalent_ui/result_webhook.py
@@ -26,22 +26,14 @@ import requests
 import covalent_ui.app as ui_server
 from covalent._results_manager import Result
 from covalent._shared_files import logger
-from covalent._shared_files.config import get_config
+from covalent._shared_files.utils import get_ui_url
 from covalent_dispatcher._db.dispatchdb import encode_dict, extract_graph, extract_metadata
 
-DEFAULT_UI_ADDRESS = get_config("user_interface.address")
-DEFAULT_UI_PORT = get_config("user_interface.port")
-
 app_log = logger.app_log
 
 # UI server webhook for result updates
 
 
-def get_ui_url(path):
-    baseUrl = f"http://{DEFAULT_UI_ADDRESS}:{DEFAULT_UI_PORT}"
-    return f"{baseUrl}{path}"
-
-
 async def send_update(result: Result) -> None:
     """
     Signal UI server about a result update. Note that the server will expect the


Test patch:
diff --git a/tests/covalent_dispatcher_tests/_cli/service_test.py b/tests/covalent_dispatcher_tests/_cli/service_test.py
index f606af0b..65ae31c4 100644
--- a/tests/covalent_dispatcher_tests/_cli/service_test.py
+++ b/tests/covalent_dispatcher_tests/_cli/service_test.py
@@ -139,6 +139,17 @@ def test_graceful_start_when_pid_exists(mocker):
 def test_graceful_start_when_pid_absent(mocker, no_triggers_flag, triggers_only_flag):
     """Test the graceful server start function."""
 
+    config_paths = [
+        "dispatcher.cache_dir",
+        "dispatcher.results_dir",
+        "dispatcher.log_dir",
+        "user_interface.log_dir",
+        "dispatcher.db_path",
+    ]
+
+    def patched_fn(entry):
+        return entry
+
     read_pid_mock = mocker.patch("covalent_dispatcher._cli.service._read_pid")
     pid_exists_mock = mocker.patch("psutil.pid_exists", return_value=False)
     rm_pid_file_mock = mocker.patch("covalent_dispatcher._cli.service._rm_pid_file")
@@ -146,6 +157,10 @@ def test_graceful_start_when_pid_absent(mocker, no_triggers_flag, triggers_only_
         "covalent_dispatcher._cli.service._next_available_port", return_value=1984
     )
     popen_mock = mocker.patch("covalent_dispatcher._cli.service.Popen")
+    get_mock = mocker.patch(
+        "covalent._shared_files.config.ConfigManager.get", side_effect=patched_fn
+    )
+    path_mock = mocker.patch("covalent_dispatcher._cli.service.Path.__init__", return_value=None)
     click_echo_mock = mocker.patch("click.echo")
     requests_mock = mocker.patch(
         "covalent_dispatcher._cli.service.requests.get",
@@ -174,6 +189,14 @@ def test_graceful_start_when_pid_absent(mocker, no_triggers_flag, triggers_only_
             )
             assert res == 1984
 
+            path_mock_calls = path_mock.mock_calls
+            get_mock_calls = get_mock.mock_calls
+
+            for each_path in config_paths:
+                assert (mocker.call(each_path) in get_mock_calls) and (
+                    mocker.call(each_path) in path_mock_calls
+                )
+
             popen_mock.assert_called_once()
             assert popen_mock.call_args[0][0] == launch_str
 
diff --git a/tests/covalent_tests/shared_files/config_test.py b/tests/covalent_tests/shared_files/config_test.py
index 59dd7b27..32a71c86 100644
--- a/tests/covalent_tests/shared_files/config_test.py
+++ b/tests/covalent_tests/shared_files/config_test.py
@@ -58,6 +58,9 @@ def test_config_manager_init_write_update_config(
     config_keys = [
         "sdk.log_dir",
         "sdk.executor_dir",
+    ]
+
+    server_specific_config_keys = [
         "dispatcher.cache_dir",
         "dispatcher.results_dir",
         "dispatcher.log_dir",
@@ -89,7 +92,13 @@ def test_config_manager_init_write_update_config(
     path_mock_calls = path_mock.mock_calls
 
     for key in config_keys:
-        assert mocker.call(key) in get_mock_calls and path_mock_calls
+        assert (mocker.call(key) in get_mock_calls) and (mocker.call(key) in path_mock_calls)
+
+    # check if creation doesn't happen during config manager init
+    for key in server_specific_config_keys:
+        assert (mocker.call(key) not in get_mock_calls) and (
+            mocker.call(key) not in path_mock_calls
+        )
 
 
 def test_set_config_str_key(mocker):
diff --git a/tests/covalent_tests/workflow/lattice_test.py b/tests/covalent_tests/workflow/lattice_test.py
index 56a1bf2d..04ee9105 100644
--- a/tests/covalent_tests/workflow/lattice_test.py
+++ b/tests/covalent_tests/workflow/lattice_test.py
@@ -26,7 +26,7 @@ import pytest
 
 import covalent as ct
 from covalent._shared_files.defaults import DefaultMetadataValues, postprocess_prefix
-from covalent_ui.result_webhook import get_ui_url
+from covalent._shared_files.utils import get_ui_url
 
 DEFAULT_METADATA_VALUES = asdict(DefaultMetadataValues())