Repository: canonical/github-runner-operator
Pull request #540: Feat/app logging
Author: login: yhaliaw
Labels: name: Libraries: Out of sync
Created: 2025-05-05T09:39:43Z | Merged: 2025-05-15T12:31:15Z



### Overview

<!-- A high level overview of the change -->

The log of the github-runner-manager service to `/var/log/github-runner-manager/{unit-name}.log`.
The unit name needs to be in the filename as all log files are ingested to loki.

### Rationale

<!-- The reason the change is needed -->

Allow the logs to be viewed from loki. With the filename containing the unit name, the log from different units can be filtered for.

### Module Changes

<!-- Any high level changes to modules and why (Service, Observer, helper) -->

Modify the `src/manager_service.py` to setup the logging.

### Checklist

- [x] The [charm style guide](https://juju.is/docs/sdk/styleguide) was applied.
- [x] The [contributing guide](https://github.com/canonical/is-charms-contributing-guide) was applied.
- [x] The changes are compliant with [ISD054 - Managing Charm Complexity](https://discourse.charmhub.io/t/specification-isd014-managing-charm-complexity/11619)
- [x] The documentation for charmhub is updated.
- [x] The PR is tagged with appropriate label (`urgent`, `trivial`, `complex`).
- [x] The changelog is updated with changes that affects the users of the charm.

<!-- Explanation for any unchecked items above -->


Diff:
diff --git a/docs/changelog.md b/docs/changelog.md
index 7193b2cba4..a9f3c3f862 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -1,5 +1,8 @@
 # Changelog
 
+### 2025-05-09
+
+- The log rotation is done by copying the logs then truncating. There is a small chance some log might be lost during the log rotation. This affects the github-runner-manager service logs, and the reactive spawning logs.
 
 ### 2025-05-06
 
diff --git a/github-runner-manager/src/github_runner_manager/cli.py b/github-runner-manager/src/github_runner_manager/cli.py
index d146593942..66dd957f73 100644
--- a/github-runner-manager/src/github_runner_manager/cli.py
+++ b/github-runner-manager/src/github_runner_manager/cli.py
@@ -4,6 +4,7 @@
 """The CLI entrypoint for github-runner-manager application."""
 
 import logging
+import sys
 from functools import partial
 from io import StringIO
 from threading import Lock
@@ -16,8 +17,6 @@
 from github_runner_manager.reconcile_service import start_reconcile_service
 from github_runner_manager.thread_manager import ThreadManager
 
-logger = logging.getLogger(__name__)
-
 
 @click.command()
 @click.option(
@@ -59,8 +58,10 @@ def main(
         port: The port to listen on the HTTP server.
         debug: Whether to start the application in debug mode.
     """
+    log_level = logging.INFO
     if debug:
-        logging.basicConfig(level=logging.DEBUG)
+        log_level = logging.DEBUG
+    logging.basicConfig(level=log_level, stream=sys.stderr)
 
     lock = Lock()
     config_str = config_file.read()
@@ -68,8 +69,8 @@ def main(
     http_server_args = FlaskArgs(host=host, port=port, debug=debug)
 
     thread_manager = ThreadManager()
-    thread_manager.add_thread(target=partial(start_reconcile_service, config, lock))
     thread_manager.add_thread(target=partial(start_http_server, config, lock, http_server_args))
+    thread_manager.add_thread(target=partial(start_reconcile_service, config, lock))
     thread_manager.start()
 
     thread_manager.raise_on_error()
diff --git a/github-runner-manager/src/github_runner_manager/http_server.py b/github-runner-manager/src/github_runner_manager/http_server.py
index ef22298861..5e04fb858e 100644
--- a/github-runner-manager/src/github_runner_manager/http_server.py
+++ b/github-runner-manager/src/github_runner_manager/http_server.py
@@ -146,6 +146,7 @@ def start_http_server(
         lock: The lock representing modification access to the managed set of runners.
         flask_args: The arguments for the flask HTTP server.
     """
+    app.logger.info("Starting the server...")
     # The lock is passed from the caller, hence the need to update the global variable.
     global _lock  # pylint: disable=global-statement
     _lock = lock
diff --git a/github-runner-manager/src/github_runner_manager/reconcile_service.py b/github-runner-manager/src/github_runner_manager/reconcile_service.py
index b472746a48..8c192fc7c0 100644
--- a/github-runner-manager/src/github_runner_manager/reconcile_service.py
+++ b/github-runner-manager/src/github_runner_manager/reconcile_service.py
@@ -18,11 +18,9 @@ def start_reconcile_service(_: ApplicationConfiguration, lock: Lock) -> None:
     Args:
         lock: The lock representing modification access to the managed set of runners.
     """
-    # The reconcile service is not implemented yet, current logging the lock status.
+    logger.info("Starting the reconcile_service...")
     while True:
-        logger.info("Lock locked: %s", lock.locked())
-        logger.info("Attempting to acquire lock for reconcile...")
         with lock:
-            logger.info("Reconciling the runners...")
+            logger.info("Acquired the lock for reconciling")
             sleep(10)
-        logger.info("Reconcile of runners completed")
+        logger.info("Released the lock")
diff --git a/github-runner-manager/tests/integration/conftest.py b/github-runner-manager/tests/integration/conftest.py
index 4f4898fb54..74af5687c5 100644
--- a/github-runner-manager/tests/integration/conftest.py
+++ b/github-runner-manager/tests/integration/conftest.py
@@ -90,7 +90,7 @@ def config_file_fixture(tmp_path_factory, config: dict) -> Path:
 
 @pytest.fixture(name="install_app", scope="module")
 def install_app_fixture() -> None:
-    subprocess.check_call([sys.executable, "-m", "pip", "install", "."])
+    subprocess.check_call([sys.executable, "-m", "pip", "install", "--force-reinstall", "."])
 
 
 @pytest.fixture(name="app", scope="function")
diff --git a/github-runner-manager/tests/integration/helper.py b/github-runner-manager/tests/integration/helper.py
index 2f985760d8..80600df185 100644
--- a/github-runner-manager/tests/integration/helper.py
+++ b/github-runner-manager/tests/integration/helper.py
@@ -5,6 +5,7 @@
 
 import subprocess
 from pathlib import Path
+from time import sleep
 from typing import Sequence
 
 import requests
@@ -50,15 +51,13 @@ def start_app(config_file: Path, extra_args: Sequence[str]) -> subprocess.Popen:
         stderr=subprocess.PIPE,
     )
     assert process.stderr is not None, "Test setup failure: Missing stderr stream"
-    logs = b""
-    for line in process.stderr:
-        if b"Address already in use" in line:
-            assert False, "Test setup failure: Port used for testing taken"
-        if b"Press CTRL+C to quit" in line:
-            break
-        logs += line
-    else:
-        assert False, f"Test setup failure: Abnormal app exit with logs:\n{logs.decode('utf-8')}"
+    for _ in range(6):
+        try:
+            health_check()
+        except (AssertionError, requests.ConnectionError):
+            sleep(10)
+            continue
+        break
     return process
 
 
diff --git a/github-runner-manager/tests/integration/test_cli.py b/github-runner-manager/tests/integration/test_cli.py
index 2f5edb5876..086981e62f 100644
--- a/github-runner-manager/tests/integration/test_cli.py
+++ b/github-runner-manager/tests/integration/test_cli.py
@@ -18,9 +18,8 @@ def test_app_reconcile(app: subprocess.Popen):
     sleep(15)
     app.terminate()
     logs = get_app_log(app)
-    assert "Attempting to acquire lock for reconcile..." in logs
-    assert "Reconciling the runners..." in logs
-    assert "Reconcile of runners completed" in logs
+    assert "Starting the server..." in logs
+    assert "Starting the reconcile_service..." in logs
 
 
 def test_app_http_server(app: subprocess.Popen):
diff --git a/src/logrotate.py b/src/logrotate.py
index 18cd7e8ed3..ebe501c779 100644
--- a/src/logrotate.py
+++ b/src/logrotate.py
@@ -11,6 +11,7 @@
 from pydantic import BaseModel
 
 from errors import LogrotateSetupError
+from manager_service import GITHUB_RUNNER_MANAGER_SERVICE_LOG_DIR
 
 LOG_ROTATE_TIMER_SYSTEMD_SERVICE = "logrotate.timer"
 
@@ -73,6 +74,15 @@ class LogrotateConfig(BaseModel):
     frequency=LogrotateFrequency.DAILY,
 )
 
+GITHUB_RUNNER_MANAGER_CONFIG = LogrotateConfig(
+    name="github-runner-manager",
+    log_path_glob_pattern=f"{GITHUB_RUNNER_MANAGER_SERVICE_LOG_DIR}/*.log",
+    rotate=0,
+    create=False,
+    notifempty=False,
+    frequency=LogrotateFrequency.DAILY,
+)
+
 
 def setup() -> None:
     """Enable and configure logrotate.
@@ -110,6 +120,7 @@ def _configure() -> None:
     """Configure logrotate."""
     _write_config(REACTIVE_LOGROTATE_CONFIG)
     _write_config(METRICS_LOGROTATE_CONFIG)
+    _write_config(GITHUB_RUNNER_MANAGER_CONFIG)
 
 
 def _write_config(logrotate_config: LogrotateConfig) -> None:
@@ -123,6 +134,7 @@ def _write_config(logrotate_config: LogrotateConfig) -> None:
         f"""{logrotate_config.log_path_glob_pattern} {{
 {logrotate_config.frequency}
 rotate {logrotate_config.rotate}
+copytruncate
 missingok
 {"notifempty" if logrotate_config.notifempty else "ifempty"}
 {"create" if logrotate_config.create else "nocreate"}
diff --git a/src/manager_service.py b/src/manager_service.py
index e7a284ca17..a26ad94e3f 100644
--- a/src/manager_service.py
+++ b/src/manager_service.py
@@ -35,6 +35,7 @@
 GITHUB_RUNNER_MANAGER_PACKAGE_PATH = "./github-runner-manager"
 JOB_MANAGER_PACKAGE_PATH = "./jobmanager/client"
 GITHUB_RUNNER_MANAGER_SERVICE_NAME = "github-runner-manager"
+GITHUB_RUNNER_MANAGER_SERVICE_LOG_DIR = Path("/var/log/github-runner-manager")
 
 _INSTALL_ERROR_MESSAGE = "Unable to install github-runner-manager package from source"
 _SERVICE_SETUP_ERROR_MESSAGE = "Unable to enable or start the github-runner-manager application"
@@ -53,7 +54,10 @@ def setup(state: CharmState, app_name: str, unit_name: str) -> None:
     """
     config = create_application_configuration(state, app_name, unit_name)
     config_file = _setup_config_file(config)
-    _setup_service_file(config_file)
+    GITHUB_RUNNER_MANAGER_SERVICE_LOG_DIR.mkdir(parents=True, exist_ok=True)
+    log_file_path = _get_log_file_path(unit_name)
+    log_file_path.touch(exist_ok=True)
+    _setup_service_file(config_file, log_file_path)
     _enable_service()
 
 
@@ -97,6 +101,19 @@ def install_package() -> None:
         raise RunnerManagerApplicationInstallError(_INSTALL_ERROR_MESSAGE) from err
 
 
+def _get_log_file_path(unit_name: str) -> Path:
+    """Get the log file path.
+
+    Args:
+        unit_name: The Juju unit name.
+
+    Returns:
+        The path to the log file.
+    """
+    log_name = unit_name.replace("/", "-") + ".log"
+    return GITHUB_RUNNER_MANAGER_SERVICE_LOG_DIR / log_name
+
+
 def _enable_service() -> None:
     """Enable the github runner manager service.
 
@@ -127,11 +144,12 @@ def _setup_config_file(config: ApplicationConfiguration) -> Path:
     return path
 
 
-def _setup_service_file(config_file: Path) -> None:
+def _setup_service_file(config_file: Path, log_file: Path) -> None:
     """Configure the systemd service.
 
     Args:
         config_file: The configuration file for the service.
+        log_file: The file location to store the logs.
     """
     service_file_content = textwrap.dedent(
         f"""\
@@ -145,6 +163,8 @@ def _setup_service_file(config_file: Path) -> None:
         ExecStart=github-runner-manager --config-file {str(config_file)} --host \
 {GITHUB_RUNNER_MANAGER_ADDRESS} --port {GITHUB_RUNNER_MANAGER_PORT}
         Restart=on-failure
+        StandardOutput=append:{log_file}
+        StandardError=append:{log_file}
 
         [Install]
         WantedBy=multi-user.target
diff --git a/tests/integration/helpers/common.py b/tests/integration/helpers/common.py
index 13d4c71a3a..be459a2185 100644
--- a/tests/integration/helpers/common.py
+++ b/tests/integration/helpers/common.py
@@ -32,6 +32,7 @@
     TEST_MODE_CONFIG_NAME,
     TOKEN_CONFIG_NAME,
 )
+from manager_service import _get_log_file_path
 from tests.status_name import ACTIVE
 
 DISPATCH_TEST_WORKFLOW_FILENAME = "workflow_dispatch_test.yaml"
@@ -417,3 +418,26 @@ async def get_file_content(unit: Unit, filepath: pathlib.Path) -> str:
     assert stdout is not None, f"Failed to get content of {filepath}, no stdout message"
     logging.info("File content of %s: %s", filepath, stdout)
     return stdout.strip()
+
+
+async def get_github_runner_manager_service_log(unit: Unit) -> str:
+    """Get the logs of github-runner-manager service.
+
+    Args:
+        unit: The unit to get the logs from.
+
+    Returns:
+        The logs.
+    """
+    log_file_path = _get_log_file_path(unit.name)
+    return_code, stdout, stderr = await run_in_unit(
+        unit,
+        f"cat {log_file_path}",
+        timeout=60,
+        assert_on_failure=True,
+        assert_msg="Failed to get the GitHub runner manager logs",
+    )
+
+    assert return_code == 0, f"Get log with cat failed with: {stderr}"
+    assert stdout is not None
+    return stdout
diff --git a/tests/integration/test_charm_no_runner.py b/tests/integration/test_charm_no_runner.py
index eefe44ac54..e715bff086 100644
--- a/tests/integration/test_charm_no_runner.py
+++ b/tests/integration/test_charm_no_runner.py
@@ -3,6 +3,7 @@
 
 """Integration tests for github-runner charm with no runner."""
 import logging
+from asyncio import sleep
 
 import pytest
 from juju.application import Application
@@ -10,7 +11,12 @@
 
 from charm_state import BASE_VIRTUAL_MACHINES_CONFIG_NAME
 from manager_service import GITHUB_RUNNER_MANAGER_SERVICE_NAME
-from tests.integration.helpers.common import reconcile, run_in_unit, wait_for
+from tests.integration.helpers.common import (
+    get_github_runner_manager_service_log,
+    reconcile,
+    run_in_unit,
+    wait_for,
+)
 from tests.integration.helpers.openstack import OpenStackInstanceHelper
 
 logger = logging.getLogger(__name__)
@@ -94,12 +100,17 @@ async def test_manager_service_started(
 ) -> None:
     """
     arrange: A working application with no runners.
-    act: Check the github runner manager service.
-    assert: The service should be running.
+    act:
+        1. Check the github runner manager service.
+        2. Force a logrotate
+    assert:
+        1. The service should be running, and logs generated.
+        2. New lines of log should be found, the initialize logs should not be found.
     """
     app = app_no_runner
     unit = app.units[0]
 
+    # 1.
     await run_in_unit(
         unit,
         f"sudo systemctl status {GITHUB_RUNNER_MANAGER_SERVICE_NAME}",
@@ -107,3 +118,25 @@ async def test_manager_service_started(
         assert_on_failure=True,
         assert_msg="GitHub runner manager service not healthy",
     )
+
+    log = await get_github_runner_manager_service_log(unit)
+    assert "Starting the server..." in log
+    assert "Starting the reconcile_service..." in log
+
+    # 2.
+    return_code, _, _ = await run_in_unit(
+        unit,
+        "sudo logrotate -f /etc/logrotate.d/github-runner-manager",
+        timeout=60,
+        assert_on_failure=True,
+        assert_msg="Failed to force rotate of logs",
+    )
+    assert return_code == 0
+
+    # Wait for more log lines.
+    await sleep(15)
+
+    log = await get_github_runner_manager_service_log(unit)
+    assert "Starting the server..." not in log
+    assert "Starting the reconcile_service..." not in log
+    assert "Acquired the lock for reconciling" in log
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
index d86625ce1a..350cca31e2 100644
--- a/tests/unit/test_charm.py
+++ b/tests/unit/test_charm.py
@@ -56,6 +56,8 @@ def side_effect_fixture(monkeypatch, tmpdir):
     monkeypatch.setattr("charm.execute_command", MagicMock())
     monkeypatch.setattr("manager_service.yaml_safe_dump", MagicMock())
     monkeypatch.setattr("manager_service.Path.expanduser", lambda x: tmpdir)
+    monkeypatch.setattr("manager_service.Path.mkdir", MagicMock())
+    monkeypatch.setattr("manager_service.Path.touch", MagicMock())
     monkeypatch.setattr("manager_service.systemd", MagicMock())
 
 
diff --git a/tests/unit/test_logrotate.py b/tests/unit/test_logrotate.py
index a06ed0382b..be44eabafa 100644
--- a/tests/unit/test_logrotate.py
+++ b/tests/unit/test_logrotate.py
@@ -110,6 +110,7 @@ def test__write_config(
     expected_logrotate_config = f"""{log_path_glob_pattern} {{
 {frequency}
 rotate {rotate}
+copytruncate
 missingok
 {"notifempty" if notifempty else "ifempty"}
 {"create" if create else "nocreate"}
diff --git a/tests/unit/test_manager_service.py b/tests/unit/test_manager_service.py
index acba6fbd83..5568d8a808 100644
--- a/tests/unit/test_manager_service.py
+++ b/tests/unit/test_manager_service.py
@@ -28,6 +28,11 @@ def mock_home_path_fixture(tmp_path_factory):
     return tmp_path_factory.mktemp("home")
 
 
+@pytest.fixture(name="mock_log_path")
+def mock_log_path_fixture(tmp_path_factory):
+    return tmp_path_factory.mktemp("log") / "github-runner-manager"
+
+
 @pytest.fixture(name="mock_systemd")
 def mock_systemd_fixture(monkeypatch):
     mock_systemd = MagicMock()
@@ -43,11 +48,12 @@ def mock_execute_command_fixture(monkeypatch):
 
 
 @pytest.fixture(name="patch_file_paths")
-def patch_file_paths(monkeypatch, mock_service_path, mock_home_path):
+def patch_file_paths(monkeypatch, mock_service_path, mock_home_path, mock_log_path):
     """Patch the file path used."""
     monkeypatch.setattr(
         "manager_service.GITHUB_RUNNER_MANAGER_SYSTEMD_SERVICE_PATH", mock_service_path
     )
+    monkeypatch.setattr("manager_service.GITHUB_RUNNER_MANAGER_SERVICE_LOG_DIR", mock_log_path)
     monkeypatch.setattr("manager_service.Path.expanduser", lambda x: mock_home_path)
 
 


Reviews:
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T08:09:07Z
user: login: yhaliaw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T02:52:56Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T05:53:41Z
user: login: yhaliaw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T05:55:41Z
user: login: yhaliaw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T05:59:40Z
user: login: yhaliaw
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T06:01:04Z
user: login: cbartz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T06:08:10Z
user: login: cbartz
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-09T06:30:05Z
user: login: javierdelapuente
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-12T13:22:32Z

Review comments:
diff_hunk: @@ -154,6 +172,8 @@ def _setup_service_file(config_file: Path) -> None:
         ExecStart=github-runner-manager --config-file {str(config_file)} --host \
 {GITHUB_RUNNER_MANAGER_ADDRESS} --port {GITHUB_RUNNER_MANAGER_PORT}
         Restart=on-failure
+        StandardOutput=append:{log_file}
+        StandardError=append:{log_file}
path: src/manager_service.py
user: login: cbartz
body: Can you check what happens on a logrotation? Will the files be written again (to avoid writing to deleted filedescriptors)
created_at: 2025-05-06T06:28:45Z
author_association: COLLABORATOR
diff_hunk: @@ -154,6 +172,8 @@ def _setup_service_file(config_file: Path) -> None:
         ExecStart=github-runner-manager --config-file {str(config_file)} --host \
 {GITHUB_RUNNER_MANAGER_ADDRESS} --port {GITHUB_RUNNER_MANAGER_PORT}
         Restart=on-failure
+        StandardOutput=append:{log_file}
+        StandardError=append:{log_file}
path: src/manager_service.py
user: login: yhaliaw
body: I have tested around, it seems the `copytruncate` option of logrotate works.
I have added a integration test for the following.
1. Check runner manager service logs for initialization lines, e.g., "Starting the server..."
2. Run `logrotate -f /etc/logrotate.d/github-runner-manager`
3. Check the log no longer include the initialization lines, but include new log lines such as "Acquired the lock..."

I think the `copytruncate` option should work for all log files (reactive, etc.). Do you see any issues with it?
created_at: 2025-05-09T02:52:56Z
author_association: COLLABORATOR
diff_hunk: @@ -18,11 +18,9 @@ def start_reconcile_service(_: ApplicationConfiguration, lock: Lock) -> None:
     Args:
         lock: The lock representing modification access to the managed set of runners.
     """
-    # The reconcile service is not implemented yet, current logging the lock status.
+    logger.info("Staring the reconcile_service...")
path: github-runner-manager/src/github_runner_manager/reconcile_service.py
user: login: cbartz
body: ```suggestion
    logger.info("Starting the reconcile_service...")
```
created_at: 2025-05-09T05:44:18Z
author_association: COLLABORATOR
diff_hunk: @@ -133,6 +134,7 @@ def _write_config(logrotate_config: LogrotateConfig) -> None:
         f"""{logrotate_config.log_path_glob_pattern} {{
 {logrotate_config.frequency}
 rotate {logrotate_config.rotate}
+copytruncate
path: src/logrotate.py
user: login: cbartz
body: Please add this change (with an explanation of the necessity) to the PR description so we can track this change in the future more easily. 
This seems to fix the current situation for reactive logs too, so might also be worth adding to the changelog as it is a bug fix.
created_at: 2025-05-09T05:45:32Z
author_association: COLLABORATOR
diff_hunk: @@ -28,12 +28,16 @@ def get_app_log(app: subprocess.Popen) -> str:
     return app.stderr.read().decode("utf-8")
 
 
-def start_app(config_file: Path, extra_args: Sequence[str]) -> subprocess.Popen:
+def start_app(
+    config_file: Path, extra_args: Sequence[str], wait_flask_start: bool = False
+) -> subprocess.Popen:
     """Start the CLI application.
 
     Args:
         config_file: The Path to the configuration file.
         extra_args: Any extra args to the CLI application.
+        wait_flask_start: Wait for flask to start before returning. Logs prior to the flask server
+            started will be removed.
path: github-runner-manager/tests/integration/helper.py
user: login: cbartz
body: How/why will the logs be removed prior to the start?
created_at: 2025-05-09T05:46:34Z
author_association: COLLABORATOR
diff_hunk: @@ -101,3 +101,17 @@ def app_fixture(
     process = start_app(config_file, [])
     yield process
     process.kill()
+
+
+@pytest.fixture(name="web_started_app", scope="function")
+def web_started_app_fixture(
path: github-runner-manager/tests/integration/conftest.py
user: login: cbartz
body: Maybe rename to `app_already_started` . Do we really need tests for both cases (app during start and already started) ?
created_at: 2025-05-09T05:49:06Z
author_association: COLLABORATOR
diff_hunk: @@ -108,15 +119,25 @@ async def test_manager_service_started(
         assert_msg="GitHub runner manager service not healthy",
     )
 
-    log_file_path = _get_log_file_path(unit.name)
-    return_code, stdout, stderr = await run_in_unit(
+    log = await get_github_runner_manager_service_log(unit)
+    assert "Starting the server..." in log
+    assert "Staring the reconcile_service..." in log
path: tests/integration/test_charm_no_runner.py
user: login: cbartz
body: ```suggestion
    assert "Starting the reconcile_service..." in log
```
created_at: 2025-05-09T05:50:14Z
author_association: COLLABORATOR
diff_hunk: @@ -28,12 +28,16 @@ def get_app_log(app: subprocess.Popen) -> str:
     return app.stderr.read().decode("utf-8")
 
 
-def start_app(config_file: Path, extra_args: Sequence[str]) -> subprocess.Popen:
+def start_app(
+    config_file: Path, extra_args: Sequence[str], wait_flask_start: bool = False
+) -> subprocess.Popen:
     """Start the CLI application.
 
     Args:
         config_file: The Path to the configuration file.
         extra_args: Any extra args to the CLI application.
+        wait_flask_start: Wait for flask to start before returning. Logs prior to the flask server
+            started will be removed.
path: github-runner-manager/tests/integration/helper.py
user: login: yhaliaw
body: The waiting for flask to start involves reading the logs until it prints a log on flask being ready.
created_at: 2025-05-09T05:55:41Z
author_association: COLLABORATOR
diff_hunk: @@ -28,12 +28,16 @@ def get_app_log(app: subprocess.Popen) -> str:
     return app.stderr.read().decode("utf-8")
 
 
-def start_app(config_file: Path, extra_args: Sequence[str]) -> subprocess.Popen:
+def start_app(
+    config_file: Path, extra_args: Sequence[str], wait_flask_start: bool = False
+) -> subprocess.Popen:
     """Start the CLI application.
 
     Args:
         config_file: The Path to the configuration file.
         extra_args: Any extra args to the CLI application.
+        wait_flask_start: Wait for flask to start before returning. Logs prior to the flask server
+            started will be removed.
path: github-runner-manager/tests/integration/helper.py
user: login: yhaliaw
body: I will change it to checking on a health endpoint. 
created_at: 2025-05-09T05:59:40Z
author_association: COLLABORATOR
diff_hunk: @@ -101,3 +101,17 @@ def app_fixture(
     process = start_app(config_file, [])
     yield process
     process.kill()
+
+
+@pytest.fixture(name="web_started_app", scope="function")
+def web_started_app_fixture(
path: github-runner-manager/tests/integration/conftest.py
user: login: yhaliaw
body: I think we do not need it.
It was due to how the waiting for flask was done. If I change the waiting for flask ready to be done by health checks instead of waiting for logs, this will not be needed.
I will be making this change.
created_at: 2025-05-09T06:01:04Z
author_association: COLLABORATOR
diff_hunk: @@ -154,6 +172,8 @@ def _setup_service_file(config_file: Path) -> None:
         ExecStart=github-runner-manager --config-file {str(config_file)} --host \
 {GITHUB_RUNNER_MANAGER_ADDRESS} --port {GITHUB_RUNNER_MANAGER_PORT}
         Restart=on-failure
+        StandardOutput=append:{log_file}
+        StandardError=append:{log_file}
path: src/manager_service.py
user: login: cbartz
body: The issue is mentioned in https://man.archlinux.org/man/logrotate.8.en#copytruncate

>Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.

But I don't think this is critical for us.
created_at: 2025-05-09T06:08:10Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-12T14:13:09Z
body: ## Test results for commit 9aa6bb08d1739f8cdfe3476093d99991122f5694

Test coverage for 9aa6bb08d1739f8cdfe3476093d99991122f5694

  ```
Wrote XML report to coverage/coverage.xml
```
  Static code analysis report

  ```
Run started:2025-05-12 13:59:31.945727

Test results:
	No issues identified.

Code scanned:
	Total lines of code: 1966
	Total lines skipped (#nosec): 3
	Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 2

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 0
		Medium: 0
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 0
		Medium: 0
		High: 0
Files skipped (0):
```
author_association: CONTRIBUTOR