Repository: NCSU-High-Powered-Rocketry-Club/AirbrakesV2
Pull request #161: Add Logger flushing
Author: login: JacksonElia
Labels: name: enhancement
description: New feature or request
Created: 2025-02-23T21:23:16Z | Merged: 2025-03-04T16:24:08Z

This needs to be tested but I cant run mocks on my laptop anymore

Also if you want me to change the launch data name I can but idrk what to change it to

Closes #160 

Linked issue #160: Make the logger flush more frequently
Due to the hard impact we had with the ground, our pi lost power. This caused us to lose a good portion of data due to our loggers internal buffering. Because the logger is on a different process, and we have a pi 5, we can afford to log more often.

Diff:
diff --git a/airbrakes/constants.py b/airbrakes/constants.py
index 0ffb19af..e8f30d5a 100644
--- a/airbrakes/constants.py
+++ b/airbrakes/constants.py
@@ -113,6 +113,8 @@ class DisplayEndingType(StrEnum):
 """The path of the folder to hold the log files in."""
 TEST_LOGS_PATH = Path("test_logs")
 """The path of the folder to hold the test log files in."""
+NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING = 100
+"""The number of lines we log before manually flushing the buffer."""
 
 STOP_SIGNAL = "STOP"
 """
diff --git a/airbrakes/mock/mock_imu.py b/airbrakes/mock/mock_imu.py
index f98890a2..e9d1199a 100644
--- a/airbrakes/mock/mock_imu.py
+++ b/airbrakes/mock/mock_imu.py
@@ -131,6 +131,7 @@ def _read_csv(
             usecols=self._needed_fields if usecols is DEFAULT else usecols,
             converters={"invalid_fields": MockIMU._convert_invalid_fields},
             skiprows=range(1, start_index + 1),
+            memory_map=True,
             **kwargs,
         )
 
diff --git a/airbrakes/telemetry/logger.py b/airbrakes/telemetry/logger.py
index 8e97fa2f..fbcd311f 100644
--- a/airbrakes/telemetry/logger.py
+++ b/airbrakes/telemetry/logger.py
@@ -15,6 +15,7 @@
     IDLE_LOG_CAPACITY,
     LOG_BUFFER_SIZE,
     MAX_GET_TIMEOUT_SECONDS,
+    NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
     STOP_SIGNAL,
 )
 from airbrakes.telemetry.packets.apogee_predictor_data_packet import ApogeePredictorDataPacket
@@ -328,6 +329,7 @@ def _logging_loop(self) -> None:  # pragma: no cover
         # Set up the csv logging in the new process
         with self.log_path.open(mode="a", newline="") as file_writer:
             writer = csv.writer(file_writer)
+            number_of_lines_logged = 0
             while True:
                 # Get a message from the queue (this will block until a message is available)
                 # Because there's no timeout, it will wait indefinitely until it gets a message.
@@ -342,3 +344,10 @@ def _logging_loop(self) -> None:  # pragma: no cover
                     if message_field == STOP_SIGNAL:
                         return
                     writer.writerow(Logger._truncate_floats(message_field))
+                    number_of_lines_logged += 1
+                    # During our Pelicanator flight, the rocket fell and had an very hard impact
+                    # causing the pi to lose power. This caused us to lose a lot of lines of data
+                    # that were not written to the log file. To prevent this from happening again,
+                    # we flush the logger 20 lines.
+                    if number_of_lines_logged % NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING == 0:
+                        file_writer.flush()
diff --git a/tests/test_components/test_logger.py b/tests/test_components/test_logger.py
index 92fd927f..6b7cae43 100644
--- a/tests/test_components/test_logger.py
+++ b/tests/test_components/test_logger.py
@@ -1,14 +1,21 @@
 import csv
 import multiprocessing
 import multiprocessing.sharedctypes
+import threading
 import time
 from collections import deque
+from functools import partial
 
 import faster_fifo
 import pytest
 from msgspec.structs import asdict
 
-from airbrakes.constants import IDLE_LOG_CAPACITY, LOG_BUFFER_SIZE, STOP_SIGNAL
+from airbrakes.constants import (
+    IDLE_LOG_CAPACITY,
+    LOG_BUFFER_SIZE,
+    NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
+    STOP_SIGNAL,
+)
 from airbrakes.telemetry.logger import Logger
 from airbrakes.telemetry.packets.apogee_predictor_data_packet import ApogeePredictorDataPacket
 from airbrakes.telemetry.packets.context_data_packet import ContextDataPacket
@@ -70,6 +77,18 @@ def convert_dict_vals_to_str(d: dict[str, float], truncation: bool = True) -> di
     return new_d
 
 
+@pytest.fixture
+def threaded_logger(monkeypatch):
+    """Modifies the Logger to run in a separate thread instead of a process."""
+    logger = Logger(LOG_PATH)
+    # Cannot use signals from child threads, so we need to monkeypatch it:
+    monkeypatch.setattr("signal.signal", lambda _, __: None)
+    target = threading.Thread(target=logger._logging_loop)
+    logger._log_process = target
+    yield logger
+    logger.stop()
+
+
 class TestLogger:
     """Tests the Logger() class in logger.py"""
 
@@ -816,6 +835,110 @@ def test_prepare_logger_packets(
             assert isinstance(logger_data_packet, LoggerDataPacket)
             assert logger_data_packet == expected
 
+    @pytest.mark.parametrize(
+        ("num_packets", "expected_flush_calls", "expected_lines_in_file"),
+        [
+            (NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING // 3, 0, 0),  # Below threshold, no flush
+            (
+                NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
+                1,
+                NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
+            ),  # At threshold, one flush
+            (
+                NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING + 5,
+                1,
+                NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
+            ),  # Above threshold, still one flush
+            (
+                2 * NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
+                2,
+                2 * NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING,
+            ),  # Two flush cycles
+        ],
+        ids=[
+            "below_flush_threshold",
+            "at_flush_threshold",
+            "above_flush_threshold",
+            "two_flush_cycles",
+        ],
+    )
+    def test_flush_called_with_monkeypatch(
+        self,
+        threaded_logger,
+        num_packets: int,
+        expected_flush_calls: int,
+        expected_lines_in_file,
+        monkeypatch,
+    ):
+        """
+        Tests that the logger calls flush() every so many lines by monkeypatching the file object's
+        flush method.
+        """
+        # Prepare sample data packets
+        context_packet = make_context_data_packet(state_letter="M")  # Avoid buffering
+        servo_packet = make_servo_data_packet(set_extension="0.0")
+        imu_data_packets = deque([make_raw_data_packet()])
+
+        flush_calls = 0
+        # Monkeypatch Path.open to return our custom TextIOWrapper
+        original_open = threaded_logger.log_path.open
+
+        def some_flush(original_flush):
+            nonlocal flush_calls
+            flush_calls += 1
+            original_flush()
+
+        def mocked_open(*args, **kwargs):
+            # Call the original open with all keyword arguments
+            file = original_open(**kwargs)
+            original_flush = file.flush
+            file.flush = partial(some_flush, original_flush)
+            return file
+
+        monkeypatch.setattr(threaded_logger.log_path.__class__, "open", mocked_open)
+
+        # Reinitialize logger to use the monkeypatched open
+        threaded_logger.start()
+        assert threaded_logger.is_running
+
+        # Log the specified number of packets
+        for _ in range(num_packets):
+            threaded_logger.log(
+                context_packet,
+                servo_packet,
+                imu_data_packets.copy(),
+                processor_data_packets=[],
+                apogee_predictor_data_packets=[],
+            )
+
+        # Give the process time to process the queue
+        time.sleep(0.1)
+
+        # Verify the number of flush calls before stop():
+        assert flush_calls == expected_flush_calls, (
+            f"Expected {expected_flush_calls} flush calls, got {flush_calls}"
+        )
+
+        # Verify the number of lines in the file before stop():
+        with threaded_logger.log_path.open() as f:
+            lines = f.readlines()
+            num_data_lines = len(lines) - 1  # Subtract header
+            assert num_data_lines == expected_lines_in_file, (
+                f"Expected {expected_lines_in_file} lines, got {num_data_lines}"
+            )
+
+        # Stop the logger cleanly to ensure all data is processed
+        threaded_logger.stop()
+        assert not threaded_logger.is_running
+
+        # Verify all packets are written
+        with threaded_logger.log_path.open() as f:
+            lines = f.readlines()
+            num_data_lines = len(lines) - 1  # Subtract header
+            assert num_data_lines == num_packets, (
+                f"Expected {num_packets} lines, got {num_data_lines}"
+            )
+
     def test_benchmark_log_method(self, benchmark, logger):
         """Tests the performance of the _prepare_logger_packets method."""
         context_packet = make_context_data_packet(state_letter="S")


Reviews:
user: login: harshil21
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-23T22:59:11Z
user: login: JacksonElia
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-02T21:32:03Z
user: login: harshil21
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-02T21:41:36Z
user: login: harshil21
body: Confirmed that this works with the test.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-04T04:16:10Z
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This PR adds periodic flushing to the CSV logger to ensure data is written reliably and adjusts tests to validate this behavior while also updating the mock IMU to use memory mapping.  
- Added a threaded_logger fixture and parameterized flush test in the logger tests  
- Modified the logger’s logging loop to flush after a set number of lines based on a new constant  
- Introduced a new constant in constants.py and updated the mock_imu to enable memory mapping

### Reviewed Changes

| File                                               | Description                                                             |
| -------------------------------------------------- | ----------------------------------------------------------------------- |
| tests/test_components/test_logger.py               | Adds a fixture and a parameterized test to check that flush is called correctly |
| airbrakes/telemetry/logger.py                       | Updates the logging loop to flush every NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING lines |
| airbrakes/constants.py                              | Introduces NUMBER_OF_LINES_TO_LOG_BEFORE_FLUSHING constant                |
| airbrakes/mock/mock_imu.py                         | Adds memory_map=True to improve CSV reading performance                   |

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.



state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T04:22:03Z
user: login: harshil21
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T04:25:23Z

Review comments:
diff_hunk: @@ -197,6 +197,7 @@ class DisplayEndingType(StrEnum):
 """The path of the folder to hold the log files in."""
 TEST_LOGS_PATH = Path("test_logs")
 """The path of the folder to hold the test log files in."""
+NUMBER_OF_LINES_TO_BEFORE_LOGGING = 20
path: airbrakes/constants.py
user: login: harshil21
body: lines to what
created_at: 2025-02-23T22:57:14Z
author_association: MEMBER
path: launch_data/pelicanator_launch_failed_pdf.csv
user: login: harshil21
body: i'm adding this in my PR which fixes the mock, so can you remove?
created_at: 2025-02-23T22:59:08Z
author_association: MEMBER
path: launch_data/pelicanator_launch_failed_pdf.csv
user: login: JacksonElia
body: lol okay u could have just branched off of this
created_at: 2025-03-02T21:32:03Z
author_association: MEMBER
path: launch_data/pelicanator_launch_failed_pdf.csv
user: login: harshil21
body: I had made one before your pr sorry lol
created_at: 2025-03-02T21:41:36Z
author_association: MEMBER
diff_hunk: @@ -816,6 +830,98 @@ def test_prepare_logger_packets(
             assert isinstance(logger_data_packet, LoggerDataPacket)
             assert logger_data_packet == expected
 
+    @pytest.mark.parametrize(
+        ("num_packets", "expected_flush_calls", "expected_lines_in_file"),
+        [
+            (19, 0, 0),  # Below threshold, no flush
+            (20, 1, 20),  # At threshold, one flush
+            (25, 1, 20),  # Above threshold, still one flush
+            (40, 2, 40),  # Two flush cycles
+        ],
+        ids=[
+            "below_flush_threshold",
+            "at_flush_threshold",
+            "above_flush_threshold",
+            "two_flush_cycles",
+        ],
+    )
+    def test_flush_called_every_20_lines_with_monkeypatch(
+        self,
+        threaded_logger,
+        num_packets: int,
+        expected_flush_calls: int,
+        expected_lines_in_file,
+        monkeypatch,
+    ):
+        """
+        Tests that the logger calls flush() every 20 lines by monkeypatching the file object's
+        flush method.
+        """
+        # Prepare sample data packets
+        context_packet = make_context_data_packet(state_letter="M")  # Avoid buffering
+        servo_packet = make_servo_data_packet(set_extension="0.0")
+        imu_data_packets = deque([make_raw_data_packet()])
+
+        flush_calls = 0
+        # Monkeypatch Path.open to return our custom TextIOWrapper
+        original_open = threaded_logger.log_path.open
+
+        def some_flush(original_flush):
+            nonlocal flush_calls
+            flush_calls += 1
+            original_flush()
+
+        def mocked_open(*args, **kwargs):
+            # Call the original open with all keyword arguments
+            file = original_open(**kwargs)
+            original_flush = file.flush
+            file.flush = partial(some_flush, original_flush)
+            return file
+
+        monkeypatch.setattr(threaded_logger.log_path.__class__, "open", mocked_open)
+
path: tests/test_components/test_logger.py
user: login: Copilot
body: Monkeypatching the entire Path class 'open' method may unintentionally affect other tests. Consider patching the 'open' method on the specific instance (threaded_logger.log_path) to limit the scope of the change.
```suggestion
        monkeypatch.setattr(threaded_logger.log_path, "open", mocked_open)
```
created_at: 2025-03-04T04:22:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -816,6 +830,98 @@ def test_prepare_logger_packets(
             assert isinstance(logger_data_packet, LoggerDataPacket)
             assert logger_data_packet == expected
 
+    @pytest.mark.parametrize(
+        ("num_packets", "expected_flush_calls", "expected_lines_in_file"),
+        [
+            (19, 0, 0),  # Below threshold, no flush
+            (20, 1, 20),  # At threshold, one flush
+            (25, 1, 20),  # Above threshold, still one flush
+            (40, 2, 40),  # Two flush cycles
+        ],
+        ids=[
+            "below_flush_threshold",
+            "at_flush_threshold",
+            "above_flush_threshold",
+            "two_flush_cycles",
+        ],
+    )
+    def test_flush_called_every_20_lines_with_monkeypatch(
+        self,
+        threaded_logger,
+        num_packets: int,
+        expected_flush_calls: int,
+        expected_lines_in_file,
+        monkeypatch,
+    ):
+        """
+        Tests that the logger calls flush() every 20 lines by monkeypatching the file object's
+        flush method.
+        """
+        # Prepare sample data packets
+        context_packet = make_context_data_packet(state_letter="M")  # Avoid buffering
+        servo_packet = make_servo_data_packet(set_extension="0.0")
+        imu_data_packets = deque([make_raw_data_packet()])
+
+        flush_calls = 0
+        # Monkeypatch Path.open to return our custom TextIOWrapper
+        original_open = threaded_logger.log_path.open
+
+        def some_flush(original_flush):
+            nonlocal flush_calls
+            flush_calls += 1
+            original_flush()
+
+        def mocked_open(*args, **kwargs):
+            # Call the original open with all keyword arguments
+            file = original_open(**kwargs)
+            original_flush = file.flush
+            file.flush = partial(some_flush, original_flush)
+            return file
+
+        monkeypatch.setattr(threaded_logger.log_path.__class__, "open", mocked_open)
+
path: tests/test_components/test_logger.py
user: login: harshil21
body: nope, you cannot modify it if you don't override the method from the class definition. It will error out.

monkeypatching does not affect other tests.
created_at: 2025-03-04T04:25:23Z
author_association: MEMBER

PR comments:
user: login: JacksonElia
created_at: 2025-03-04T02:41:44Z
body: lol why are you on all my commits now
author_association: MEMBER
user: login: JacksonElia
created_at: 2025-03-04T02:42:26Z
body: and what did u do to my branch
author_association: MEMBER
user: login: harshil21
created_at: 2025-03-04T04:16:29Z
body: > and what did u do to my branch

I made it based
author_association: MEMBER
user: login: JacksonElia
created_at: 2025-03-04T15:50:50Z
body: @harshil21 why don't you make unit tests depend on constants 😭  thats like the whole point of having constants
author_association: MEMBER
user: login: harshil21
created_at: 2025-03-04T15:55:00Z
body: woops sorry grok wrote a bit of these tests

(it failed awfully in actually mocking it though I wrote that)
author_association: MEMBER