Repository: enjoy-digital/litex (language: python)

Issue:
Add a proper CI test for the various configurations
Previously we were testing some configurations in `test_targets` (that has been removed with https://github.com/enjoy-digital/litex/commit/f31f9a20f0b9134af25e08d1d768c72bc07e473f since required rework/improvements). We should write a proper test that would verify that various configurations are building correctly and when possible run a simulation with Verilator or try to build the bitstream with the open-source toolchains (that we can install on CI).

The aim is not to cover every possible configuration but to have a quick way to detect and fix possible regressions.


Discussion / hints:
While working on this, we should make sure to test the different CPU that can be simulated with Verilator (so also cover https://github.com/enjoy-digital/litex/issues/586).
Hi @enjoy-digital ,

I was working on this same topic in my repo when I stumbled on this open issue. I have done some initial work on adding tests for checking that the CPUs work in simulation and a basic structure is working in my CI. 

I tried a couple of approaches:

1. Try to use the bios as it is and interact with it through python subprocess and wait for some string pattern to say pass/fail. But the code to run this test started becoming complicated and not very elegant (handling the interactive console).
2. Create a simple app which will just print an output like "Hello World" and then stop the simulation by calling `sim_finish()` and have this app loaded by bios. This again involved quite a bit of changes in the litex build steps.
3. Add `BIOS_TEST_MODE` as a constant in the `litex_sim.py` and if that is set then the bios would print a message and then exit the simulation. I found this to be the path of least resistance and involved minimal changes. Using this I have setup unit tests which will run the simulation on all the supported CPUs (currently only RiscV favours) and return a pass/fail. Moreover I feel this method can be extended in future to add more software based tests like CPU benchmarks etc.

I would like to know your thoughts on this and if you have any other ideas I will try them out.

Thanks
Navaneeth 
@navan93 -- I think a lot of this sounds pretty awesome but the implementation details can be hard.

On the topic of testing on real hardware, me and @antmicro are working on a few things at https://docs.google.com/document/d/1H3x34bdztWy-uN8EYe-FxJL2YojIjd-0feL9d8BMtN4/edit#heading=h.d8vkktb69rj3 and https://antmicro.com/blog/2021/08/open-source-github-actions-runners-with-gcp-and-terraform/

We are also looking at testing in and using Renode. There is an example of the work @antmicro is doing for the Zephyr project at https://zephyr-dashboard.renode.io/


Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 46c49a3f64..176ebf7ec6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -3,7 +3,7 @@ name: ci
 on: [push, pull_request]
 
 jobs:
-  build:
+  regression-test:
     runs-on: ubuntu-18.04
     steps:
       # Checkout Repository
@@ -14,9 +14,13 @@ jobs:
       - name: Install Tools
         run: |
           sudo apt-get install wget build-essential python3
+          sudo apt-get install verilator libevent-dev libjson-c-dev
           pip3 install setuptools
           pip3 install requests
           pip3 install meson
+          pip3 install ninja
+          pip3 install nmigen-yosys
+          pip3 install pexpect
 
       # Install (n)Migen / LiteX / Cores
       - name: Install LiteX
@@ -41,4 +45,5 @@ jobs:
       - name: Run Tests
         run: |
           export GITHUB_ACTIONS=1
+          export PATH=$PATH:$(echo $PWD/../riscv64-*/bin/)
           python3 setup.py test


Test patch:
diff --git a/test/test_boot.py b/test/test_boot.py
new file mode 100644
index 0000000000..9588c43a75
--- /dev/null
+++ b/test/test_boot.py
@@ -0,0 +1,60 @@
+#
+# This file is part of LiteX.
+#
+# Copyright (c) 2021 Navaneeth Bhardwaj <navan93@gmail.com>
+# SPDX-License-Identifier: BSD-2-Clause
+
+import unittest
+import pexpect
+import sys
+
+class TestCPU(unittest.TestCase):
+    def boot_test(self, cpu_type):
+        cmd = f'lxsim --cpu-type={cpu_type}'
+        litex_prompt = [b'\033\[[0-9;]+mlitex\033\[[0-9;]+m>']
+        is_success = True
+        with open("/tmp/test_boot_log", "wb") as result_file:
+            p = pexpect.spawn(cmd, timeout=None, logfile=result_file)
+            try:
+                match_id = p.expect(litex_prompt, timeout=1200)
+            except pexpect.EOF:
+                print('\n*** Premature termination')
+                is_success = False
+            except pexpect.TIMEOUT:
+                print('\n*** Timeout ')
+                is_success = False
+
+        if not is_success:
+            print(f'*** {cpu_type} Boot Failure')
+            with open("/tmp/test_boot_log", "r") as result_file:
+                print(result_file.read())
+        else:
+            p.terminate(force=True)
+            print(f'*** {cpu_type} Boot Success')
+
+        return is_success
+
+    def test_vexriscv(self):
+        self.assertTrue(self.boot_test("vexriscv"))
+
+    def test_vexriscv_smp(self):
+        self.assertTrue(self.boot_test("vexriscv_smp"))
+
+    def test_cv32e40p(self):
+        self.assertTrue(self.boot_test("cv32e40p"))
+
+    def test_ibex(self):
+        self.assertTrue(self.boot_test("ibex"))
+
+    def test_serv(self):
+        self.assertTrue(self.boot_test("serv"))
+
+    def test_femtorv(self):
+        self.assertTrue(self.boot_test("femtorv"))
+
+    def test_picorv32(self):
+        self.assertTrue(self.boot_test("picorv32"))
+
+    def test_minerva(self):
+        self.assertTrue(self.boot_test("minerva"))
+