Repository: cmu-argus-2/FSW-mainboard
Pull request #180: SIL CI Integration
Author: login: atletea
Created: 2025-02-23T19:55:33Z | Merged: 2025-03-06T17:25:42Z

- Added sil.yml file that sets up and runs the simulation for each push on Github workflow
- Added sil_run.py that runs the simulation for 30 seconds 
- Added parse_sillogs.py which will parse logs and stop the workflow from successful execution if there are errors present in the simulation run 
The parse_sillogs.py can be modified depending on what we want to error check for, for now the workflow should work for any commit/push. 


Diff:
diff --git a/.github/workflows/sil.yml b/.github/workflows/sil.yml
new file mode 100644
index 00000000..718b4c6e
--- /dev/null
+++ b/.github/workflows/sil.yml
@@ -0,0 +1,75 @@
+name: SIL CI
+
+on:
+  push:
+    branches:
+      - '*'
+  pull_request:
+    branches:
+      - '*'
+
+jobs:
+  sil-simulation:
+    name: SIL simulation
+
+    strategy:
+      fail-fast: false
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v3
+        with:
+          submodules: 'recursive'
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.9
+
+      - name: Cache pip
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Setup virtual environment
+        run: |
+          python3 -m venv .venv --system-site-packages
+          source .venv/bin/activate
+      - name: Install main repository dependencies
+        run: |
+          pip install --upgrade pip
+          pip install -r requirements.txt
+      - name: Update submodules to latest commit
+        run: |
+          git submodule init
+          git submodule update
+
+      - name: Install submodule dependencies
+        run: |
+          sh install.sh
+          echo "PYTHONPATH=$PYTHONPATH:$(pwd)" >> $GITHUB_ENV  # Add current directory to PYTHONPATH 
+          python -c "import argusim; print('argusim is installed')"
+
+      - name: Verify installation
+        run: pip list  # Confirm that brahe and argusim are installed
+      - name: Set timestamp
+        run: echo "TIMESTAMP=$(date +'%Y-%m-%d_%H-%M-%S')" >> $GITHUB_ENV
+      - name: Define Runtime
+        run: echo "RUNTIME=300" >> $GITHUB_ENV
+      - name: Define Log file name
+        run: echo "OUTFILE=sil_logs.log" >> $GITHUB_ENV
+      
+      - name: Run flight software
+        run: python3 sil_run.py --duration ${{env.RUNTIME}} --outfile ${{env.OUTFILE}}
+      
+      - name: Upload log file
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ env.TIMESTAMP }}-sil-logs
+          path: ${{env.OUTFILE}}
+      
diff --git a/flight/core/template_task.py b/flight/core/template_task.py
index f0cdcfc0..f1570fec 100644
--- a/flight/core/template_task.py
+++ b/flight/core/template_task.py
@@ -24,7 +24,7 @@ def debug(self, msg):
         :param msg: Debug message to print
         :param level: > 1 will print as a sub-level
         """
-        logger.info(f"[{self.ID}][{self.name}] {msg}")
+        logger.error(f"[{self.ID}][{self.name}] {msg}")
 
     def set_frequency(self, frequency):
         """
diff --git a/requirements.txt b/requirements.txt
index 98d7e6fc..8810f0ad 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -8,3 +8,4 @@ flake8
 black
 isort
 pre-commit
+argparse
diff --git a/sil_run.py b/sil_run.py
new file mode 100644
index 00000000..cab380ac
--- /dev/null
+++ b/sil_run.py
@@ -0,0 +1,67 @@
+import argparse
+import os
+import signal
+import subprocess
+import time
+
+DEFAULT_RUNTIME = 5 * 60  # 5 minutes
+DEFAULT_OUTFILE = "sil_logs.log"
+
+# KEYWORD SEARCHES:
+# List of all keywords to probe the log for
+# Enter as a dictionary of KEYWORD - COLOR combos
+KEYWORDS = {"WARNING": "\033[93m", "ERROR": "\033[91m"}
+
+
+def FSW_simulate(runtime: float, outfile: str) -> None:
+    try:
+        with open(outfile, "w") as log_file:
+            process = subprocess.Popen(["./run.sh", "simulate"], stdout=log_file, stderr=log_file, preexec_fn=os.setsid)
+            print(f"Running simulation for {runtime} seconds, output written to {outfile}")
+            time.sleep(runtime)
+            print("Terminating...")
+            os.killpg(os.getpgid(process.pid), signal.SIGKILL)
+
+    except Exception as e:
+        print(f"Error: {e}")
+
+
+def parse_FSW_logs(outfile):
+    errors_detected = False
+    with open(outfile, "r") as log_file:
+        for line in log_file:
+            for keyword in KEYWORDS.keys():
+                if keyword in line:
+                    print(f"{KEYWORDS[keyword]}{line}")
+                    if keyword == "ERROR":
+                        errors_detected = True
+
+    if errors_detected:
+        raise Exception("FSW Simulation Failed")
+
+
+if __name__ == "__main__":
+
+    # Define Parser
+    parser = argparse.ArgumentParser(prog="SIL_tester")
+
+    # Add arguments
+    parser.add_argument(
+        "--duration",
+        default=DEFAULT_RUNTIME,
+        help=f"Duration (in seconds) to simulate FSW for [float, default: {DEFAULT_RUNTIME}s]",
+    )
+    parser.add_argument(
+        "--outfile",
+        default=DEFAULT_OUTFILE,
+        help=f"Log file to save FSW logs to [string, default: {DEFAULT_OUTFILE}] \n NOTE: Enter filename with extension",
+    )
+
+    # Parse Arguments
+    args = parser.parse_args()
+
+    # Run script
+    FSW_simulate(int(args.duration), args.outfile)
+
+    # Parse Logs
+    parse_FSW_logs(args.outfile)


Reviews:
user: login: Ibrassow
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-02-24T01:10:01Z
user: login: kmkar31
body: Agree with Ibrahima's comments. Nothing else to add
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-02-24T01:28:35Z
user: login: Ibrassow
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-24T01:40:21Z
user: login: kmkar31
body: Since my changes are the same as Ibrahima's, I'll approve so I don't block a merge
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-02-24T14:12:54Z
user: login: kmkar31
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T17:56:12Z
user: login: kmkar31
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T18:02:55Z
user: login: kmkar31
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T18:03:01Z
user: login: kmkar31
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T18:03:34Z
user: login: kmkar31
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T19:52:19Z
user: login: Ibrassow
body: Great
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-06T08:22:25Z
user: login: kmkar31
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-06T17:20:29Z

Review comments:
diff_hunk: @@ -0,0 +1,19 @@
+log_filename = "sil_sim.log"
+error_filename = "sil_err.log"
+
+python_err = False
+sim_err = False
+with open(log_filename, "r") as log_file, open(error_filename, "w") as error_log:
+    for line in log_file:
+        # errors running the SIL
+        if "Traceback" in line:
+            python_err = True
+        if "[ERROR]" in line:
+            sim_err = True
+        if python_err or sim_err:
+            error_log.write(line)
+
+if python_err:
+    raise Exception("Python errors found, see error log.")
+if sim_err:
+    raise Exception("Sim errors found, see error log.")
path: parse_sillogs.py
user: login: Ibrassow
body: - For both python scripts, put everything under a if __ name __ == "__ main __": ... 
- For the configuration, use command line arguments: time duration, filename, etc so we can generalize those scripts for other purposes. 
created_at: 2025-02-24T01:07:53Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+import os
+import signal
+import subprocess
+import time
+
+log_filename = "sil_sim.log"
+# error_log_filename = f"{timestamp}_err.log"
+
+try:
+    with open(log_filename, "w") as log_file:
+        process = subprocess.Popen(["./run.sh", "simulate"], stdout=log_file, stderr=log_file, preexec_fn=os.setsid)
+        print(f"Running simulation for 120 seconds, output written to {log_filename}")
+        time.sleep(30)
path: sil_run.py
user: login: Ibrassow
body: see comment above
created_at: 2025-02-24T01:08:04Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+log_filename = "sil_sim.log"
+error_filename = "sil_err.log"
+
+python_err = False
+sim_err = False
+with open(log_filename, "r") as log_file, open(error_filename, "w") as error_log:
+    for line in log_file:
+        # errors running the SIL
+        if "Traceback" in line:
+            python_err = True
+        if "[ERROR]" in line:
+            sim_err = True
+        if python_err or sim_err:
+            error_log.write(line)
path: parse_sillogs.py
user: login: Ibrassow
body: you can have those keywords defined in a config file (.txt or whatever) that your scripts will basically read to know all triggers (e.g. so people could add "WARNING" to the list easily)
created_at: 2025-02-24T01:09:20Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,77 @@
+name: SIL CI
+
+on:
+  push:
+    branches:
+      - '*'
+  pull_request:
+    branches:
+      - '*'
+
+jobs:
+  sil-simulation:
+    name: SIL simulation
+
+    strategy:
+      fail-fast: false
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v3
+        with:
+          submodules: 'recursive'
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.9
+
+      - name: Cache pip
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Setup virtual environment
+        run: |
+          python3 -m venv .venv --system-site-packages
+          source .venv/bin/activate
+      - name: Install main repository dependencies
+        run: |
+          pip install --upgrade pip
+          pip install -r requirements.txt
+      - name: Update submodules to latest commit
+        run: |
+          git submodule init
+          git submodule update
+
+      - name: Install submodule dependencies
+        run: |
+          sh install.sh
+          echo "PYTHONPATH=$PYTHONPATH:$(pwd)" >> $GITHUB_ENV  # Add current directory to PYTHONPATH 
+          python -c "import argusim; print('argusim is installed')"
+
+      - name: Verify installation
+        run: pip list  # Confirm that brahe and argusim are installed
+      - name: Set timestamp
+        run: echo "TIMESTAMP=$(date +'%Y-%m-%d_%H-%M-%S')" >> $GITHUB_ENV
+      - name: Run flight software 30 sec
+        run: python3 sil_run.py 
+      - name: Upload log file
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ env.TIMESTAMP }}-sil-logs
+          path: sil_sim.log
+      - name: Parse logs for errors
+        run: python3 parse_sillogs.py
+      - name: Upload error log file
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: ${{ env.TIMESTAMP }}-sil-errs
+          path: sil_err.log
path: .github/workflows/sil.yml
user: login: Ibrassow
body: add some spacing between the job names for clarity
created_at: 2025-02-24T01:09:51Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+import os
+import signal
+import subprocess
+import time
+
+log_filename = "sil_sim.log"
+# error_log_filename = f"{timestamp}_err.log"
+
+try:
+    with open(log_filename, "w") as log_file:
+        process = subprocess.Popen(["./run.sh", "simulate"], stdout=log_file, stderr=log_file, preexec_fn=os.setsid)
+        print(f"Running simulation for 120 seconds, output written to {log_filename}")
+        time.sleep(30)
path: sil_run.py
user: login: kmkar31
body: At the current frequency (GPS runs at 0.03 Hz) , I'd rather run this script for ~150s or so
created_at: 2025-02-24T01:27:58Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+import os
+import signal
+import subprocess
+import time
+
+log_filename = "sil_sim.log"
+# error_log_filename = f"{timestamp}_err.log"
+
+try:
+    with open(log_filename, "w") as log_file:
+        process = subprocess.Popen(["./run.sh", "simulate"], stdout=log_file, stderr=log_file, preexec_fn=os.setsid)
+        print(f"Running simulation for 120 seconds, output written to {log_filename}")
+        time.sleep(30)
path: sil_run.py
user: login: Ibrassow
body: good point 

Honestly, @atletea go for > 5min
created_at: 2025-02-24T01:40:21Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+log_filename = "sil_sim.log"
+error_filename = "sil_err.log"
+
+python_err = False
+sim_err = False
+with open(log_filename, "r") as log_file, open(error_filename, "w") as error_log:
+    for line in log_file:
+        # errors running the SIL
+        if "Traceback" in line:
+            python_err = True
+        if "[ERROR]" in line:
+            sim_err = True
+        if python_err or sim_err:
+            error_log.write(line)
+
+if python_err:
+    raise Exception("Python errors found, see error log.")
+if sim_err:
+    raise Exception("Sim errors found, see error log.")
path: parse_sillogs.py
user: login: kmkar31
body: Fixed
created_at: 2025-03-05T17:56:12Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,77 @@
+name: SIL CI
+
+on:
+  push:
+    branches:
+      - '*'
+  pull_request:
+    branches:
+      - '*'
+
+jobs:
+  sil-simulation:
+    name: SIL simulation
+
+    strategy:
+      fail-fast: false
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v3
+        with:
+          submodules: 'recursive'
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.9
+
+      - name: Cache pip
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Setup virtual environment
+        run: |
+          python3 -m venv .venv --system-site-packages
+          source .venv/bin/activate
+      - name: Install main repository dependencies
+        run: |
+          pip install --upgrade pip
+          pip install -r requirements.txt
+      - name: Update submodules to latest commit
+        run: |
+          git submodule init
+          git submodule update
+
+      - name: Install submodule dependencies
+        run: |
+          sh install.sh
+          echo "PYTHONPATH=$PYTHONPATH:$(pwd)" >> $GITHUB_ENV  # Add current directory to PYTHONPATH 
+          python -c "import argusim; print('argusim is installed')"
+
+      - name: Verify installation
+        run: pip list  # Confirm that brahe and argusim are installed
+      - name: Set timestamp
+        run: echo "TIMESTAMP=$(date +'%Y-%m-%d_%H-%M-%S')" >> $GITHUB_ENV
+      - name: Run flight software 30 sec
+        run: python3 sil_run.py 
+      - name: Upload log file
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ env.TIMESTAMP }}-sil-logs
+          path: sil_sim.log
+      - name: Parse logs for errors
+        run: python3 parse_sillogs.py
+      - name: Upload error log file
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: ${{ env.TIMESTAMP }}-sil-errs
+          path: sil_err.log
path: .github/workflows/sil.yml
user: login: kmkar31
body: Done
created_at: 2025-03-05T18:02:55Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+import os
+import signal
+import subprocess
+import time
+
+log_filename = "sil_sim.log"
+# error_log_filename = f"{timestamp}_err.log"
+
+try:
+    with open(log_filename, "w") as log_file:
+        process = subprocess.Popen(["./run.sh", "simulate"], stdout=log_file, stderr=log_file, preexec_fn=os.setsid)
+        print(f"Running simulation for 120 seconds, output written to {log_filename}")
+        time.sleep(30)
path: sil_run.py
user: login: kmkar31
body: Done
created_at: 2025-03-05T18:03:01Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+import os
+import signal
+import subprocess
+import time
+
+log_filename = "sil_sim.log"
+# error_log_filename = f"{timestamp}_err.log"
+
+try:
+    with open(log_filename, "w") as log_file:
+        process = subprocess.Popen(["./run.sh", "simulate"], stdout=log_file, stderr=log_file, preexec_fn=os.setsid)
+        print(f"Running simulation for 120 seconds, output written to {log_filename}")
+        time.sleep(30)
path: sil_run.py
user: login: kmkar31
body: Done
created_at: 2025-03-05T18:03:34Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,19 @@
+log_filename = "sil_sim.log"
+error_filename = "sil_err.log"
+
+python_err = False
+sim_err = False
+with open(log_filename, "r") as log_file, open(error_filename, "w") as error_log:
+    for line in log_file:
+        # errors running the SIL
+        if "Traceback" in line:
+            python_err = True
+        if "[ERROR]" in line:
+            sim_err = True
+        if python_err or sim_err:
+            error_log.write(line)
path: parse_sillogs.py
user: login: kmkar31
body: Done. I have a dictionary inside sil_run.py that has different keywords and mapping to colors in the log file. Additioanl keywords can be added there. There is specific handling for 'ERROR' alone that throws an exception and causes the CI test to fail
created_at: 2025-03-05T19:52:19Z
author_association: COLLABORATOR

PR comments:
user: login: Akshat412
created_at: 2025-02-28T21:47:04Z
body: My changes are also the same as Ibrahima's. Everything else looks good!

For actually testing FSW with this, we can temporarily add a try catch in every task, essentially some method to raise every error in FSW as an ERROR or WARNING. IDK if there's a clean way of doing this but it should be done periodically to fully utilize this script.
author_association: COLLABORATOR
user: login: kmkar31
created_at: 2025-03-05T18:22:02Z
body: > My changes are also the same as Ibrahima's. Everything else looks good!
> 
> For actually testing FSW with this, we can temporarily add a try catch in every task, essentially some method to raise every error in FSW as an ERROR or WARNING. IDK if there's a clean way of doing this but it should be done periodically to fully utilize this script.

I'm out of ideas on where to add a try except. I've checked adding except handling at ScheduledTask and inside Scheduler, but both don't do it
author_association: COLLABORATOR
user: login: Akshat412
created_at: 2025-03-05T18:37:00Z
body: What about individually within each task, inside the task's main function?
author_association: COLLABORATOR
user: login: kmkar31
created_at: 2025-03-05T19:02:06Z
body: Thanks for the idea. Changed the debug process inside TemplateTask to throw log.error instead of log.info. This should not affect flight ops since debug will be False then.

SIDE NOTE: There is an error inside the Command task which now throws an error and therefore the SIL CI test fails here
author_association: COLLABORATOR
user: login: Ibrassow
created_at: 2025-03-06T08:23:24Z
body: > Thanks for the idea. Changed the debug process inside TemplateTask to throw log.error instead of log.info. This should not affect flight ops since debug will be False then.
> 
> SIDE NOTE: There is an error inside the Command task which now throws an error and therefore the SIL CI test fails here

I assume this is fixed?
author_association: COLLABORATOR
user: login: kmkar31
created_at: 2025-03-06T11:09:06Z
body: Yep, all good now
author_association: COLLABORATOR