Repository: xsuite/xsuite (language: python)

Issue:
Run ocl tests separately
## Description

When running Xtrack tests on `ContextPyopencl`, run each file separately to avoid the `OUT_OF_HOST_MEMORY` OpenCL error, whose cause is still unclear.


Discussion / hints:

Fix patch:
diff --git a/Dockerfile b/Dockerfile
index f50099f9..7ca0824a 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -46,5 +46,7 @@ RUN pip install --upgrade cython pytest pyopencl gitpython \
     done
 
 # Don't run tests from /opt/venv not to confuse imports
+COPY run_tests.sh /opt/
+RUN chmod +x /opt/run_tests.sh
 WORKDIR /opt
 CMD python3 /opt/xsuite/xtrack/examples/print_package_paths.py


Test patch:
diff --git a/.github/workflows/test_gh.yaml b/.github/workflows/test_gh.yaml
index bfecc800..c3f6133a 100644
--- a/.github/workflows/test_gh.yaml
+++ b/.github/workflows/test_gh.yaml
@@ -64,6 +64,14 @@ jobs:
 
     # Steps represent a sequence of tasks that will be executed as part of the job
     steps:
+    - name: Increase swapfile
+      run: |
+        sudo swapoff -a
+        sudo fallocate -l 15G /swapfile
+        sudo chmod 600 /swapfile
+        sudo mkswap /swapfile
+        sudo swapon /swapfile
+        sudo swapon --show
     - name: Setup Miniforge
       uses: conda-incubator/setup-miniconda@v2
       with:
diff --git a/.github/workflows/test_gpu.yaml b/.github/workflows/test_gpu.yaml
index 1587315c..62d83a04 100644
--- a/.github/workflows/test_gpu.yaml
+++ b/.github/workflows/test_gpu.yaml
@@ -111,10 +111,18 @@ jobs:
         image_id: ${{ needs.build-test-image.outputs.image_id }}
         test_contexts: ${{ inputs.test_contexts }}
       run: |
+        mkdir -p reports/${{ matrix.test-suite }}
         docker run --rm --gpus all \
           --env XOBJECTS_TEST_CONTEXTS="${test_contexts}" \
+          -v $PWD/reports/${{ matrix.test-suite }}:/opt/reports:Z \
           ${image_id} \
-          pytest --color=yes -v /opt/xsuite/${{ matrix.test-suite }}/tests
+          /opt/run_tests.sh /opt/xsuite/${{ matrix.test-suite }}/tests
+    - name: Upload report
+      uses: actions/upload-artifact@v3
+      with:
+        name: report-${{ matrix.test-suite }}
+        path: reports/${{ matrix.test-suite }}/report.html
+        if-no-files-found: ignore
 
   # Cleanup after the tests by removing the image and making sure there are
   # no unused images and stopped containers
diff --git a/docs/testing.rst b/docs/testing.rst
index 09a47cda..524b8f57 100644
--- a/docs/testing.rst
+++ b/docs/testing.rst
@@ -11,13 +11,12 @@ dependencies need to be installed. These are Docker and Nvidia drivers and
 Container Toolkit (assuming the test machine uses an Nvidia GPU). The steps
 to accomplish this are listed below.
 
-Setup of the test runner machine
-================================
+Setup of the test runner machine (Ubuntu)
+=========================================
 
 An OpenStack GPU-capable machine
 --------------------------------
 
-We use Ubuntu, because Nvidia does not provide drivers for CS8.
 This can be set up largely following the guide
 `here <https://abpcomputing.web.cern.ch/guides/openstackUbuntu/>`_.
 
@@ -98,3 +97,221 @@ Afterwards, we install and run the runner as a service with user `ubuntu`:
 
     ./svc.sh install ubuntu
     ./svc.sh start
+
+
+Setup of the test runner machine (Alma 8)
+=========================================
+
+Synopsis
+--------
+
+On the AlmaLinux 8 virtual machine (the “host”) will be running a GitHub
+runner executing Xsuite tests in a containerised environment. In order
+to support GPU execution contexts, Nvidia drivers and the Nvidia
+Container Toolkit will need to be installed. At the time of writing this
+guide, the Nvidia guide states that Docker is not supported under RHEL
+8/CentOS 8 (and so effectively Alma 8 as well), and that is why we will
+use Podman instead of Docker. Podman is a container environment similar
+to Docker, however it does not require a separate daemon to run
+containers, which makes it more lighweight.
+
+Setup a user account
+--------------------
+
+We can set up an appropriate GPU-capable OpenStack VM in the same way as
+in the previous section (Ubuntu), or simply by following the GUI wizard
+on openstack.cern.ch.
+
+On the fresh Alma VM we first set up a user account which we will use to
+run our actions:
+
+.. code:: bash
+
+   adduser xsuite
+
+We add the user to the sudoers file, by appending the line
+``xsuite  ALL=(ALL)   NOPASSWD:ALL`` to ``/etc/sudoers``. If necessary,
+copy the authorised SSH key from the root account
+(``/root/.ssh/authorized_keys``) to the new account
+(``/home/xsuite/.ssh/authorized_keys``). Fix permissions
+(``sudo chown -R xsuite:xsuite .ssh`` and ``chmod -R +rw .ssh``).
+
+Installing Nvidia drivers
+-------------------------
+
+We will largely be following the official Nvidia guide [1]_, however
+only as far as installing the drivers. CUDA is not necessary on the host
+machine, only inside the containers.
+
+First, some prerequisites are necessary. In this guide, we will be
+installing the drivers using the “network RPM” method in Nvidia’s guide.
+We will perform a DKMS installation, so that the drivers get recompiled
+whenever there is a kernel update, so that it does not need to be done
+manually. To this end, we need to install kernel headers:
+
+.. code:: bash
+
+   sudo dnf install kernel-devel-$(uname -r) kernel-headers-$(uname -r)
+
+To satisfy other requirements of the Nvidia driver package, we enable
+the third party repository EPEL:
+
+.. code:: bash
+
+   sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
+
+Then we can enable the network repo and install the drivers:
+
+.. code:: bash
+
+   sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
+   sudo dnf clean all
+   sudo dnf -y module install nvidia-driver:latest-dkms
+   sudo dnf clean expire-cache  # clean dnf cache afterwards
+
+To check if everything works, we can ``sudo reboot``, then run the
+following command, which, if all went well, should return a summary of
+the available GPUs:
+
+.. code:: bash
+
+   nvidia-smi
+
+..
+
+   **Troubleshooting Note:** If at this stage the driver is not working,
+   it could be that it was not picked up by DKMS. We can verify this by
+   running ``dkms status``: if there is no ``nvidia/...`` entry, or if
+   to the right of it its status is not listed as ``installed``, we can
+   run ``dkms autoinstall`` to attempt to recompile the drivers.
+
+Installing the Nvidia Container Toolkit
+---------------------------------------
+
+We will follow the instruction of the official Nvidia guide [2]_, the
+steps of which are summarised below.
+
+A container environment is a prerequisite for installing the NCT. We can
+easily install Podman with:
+
+.. code:: bash
+
+   sudo dnf install podman
+
+Podman is compatible with the Container Device Interface specification,
+which means that only the base components of the Nvidia Container
+Toolkit are needed. We install the required package:
+
+.. code:: bash
+
+   sudo dnf clean expire-cache
+   sudo dnf install -y nvidia-container-toolkit-base
+
+Check that it works:
+
+.. code:: bash
+
+   nvidia-ctk --version
+
+And generate the CDI specification with:
+
+.. code:: bash
+
+   sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
+
+To be able to run rootless containers with ``podman``, we change the
+following configuration:
+
+.. code:: bash
+
+   sudo sed -i 's/^#no-cgroups = false/no-cgroups = true/;' /etc/nvidia-container-runtime/config.toml
+
+When running rootless, we may also encounter permission issues with
+SELinux. We need to add an appropriate label ``container_file_t`` to the
+Nvidia device files:
+
+.. code:: bash
+
+   sudo semanage fcontext -a -t container_file_t '/dev/nvidia.*'
+   restorecon -v /dev/*
+
+Check that everything works with:
+
+.. code:: bash
+
+   podman run --rm --gpus all cupy/cupy:latest nvidia-smi
+
+Finally, we make a link called ``docker`` pointing to ``podman``, so
+that the workflows (which presume Docker) work on the new machine:
+
+.. code:: bash
+
+   sudo ln -s /usr/bin/podman /usr/bin/docker
+
+Setup the GitHub runner
+-----------------------
+
+Navigate to *Settings > Actions > Runners* on GitHub and follow the
+instructions for creating the new runner. Once this is done, there are
+three final steps that need to be done before we enable the runner
+service.
+
+Configure SELinux
+~~~~~~~~~~~~~~~~~
+
+We need to label the service script as an executable to SELinux,
+otherwise it will prevent us from launching the service.
+
+.. code:: bash
+
+   sudo semanage fcontext -a -t bin_t '/home/xsuite/actions-runner/runsvc.sh'
+   sudo semanage fcontext -a -t bin_t '/home/xsuite/actions-runner/bin(/.*)?'
+   sudo restorecon -v -R /home/xsuite/actions-runner/
+
+Set the right container format
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since we are using a docker Dockerfile format, which is slightly
+different to the OCI format, to which podman defaults, we need to change
+the setting for podman to use the Docker format. To achieve this, we add
+an environment variable to the runner service file:
+
+.. code:: bash
+
+   sudo ./svc.sh install xsuite && sudo ./svc.sh stop  # create but don't start the service
+   sudo systemctl edit actions.runner.xsuite-xsuite.xsuite-alma-tests.service
+
+In the opened editor (which may be empty), we paste the following:
+
+.. code:: ini
+
+   [Service]
+   Environment="BUILDAH_FORMAT=docker"
+
+Enable account lingering
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order for Podman to be able to function headless, we need to enable
+account lingering, as otherwise, systemd will kill any user process when
+there is no login session for the user.
+
+.. code:: bash
+
+   sudo loginctl enable-linger xsuite
+
+Enable and start the service
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Finally, we can start the runner service, which will immediately begin
+listening for new jobs:
+
+.. code:: bash
+
+   sudo ./svc.sh install xsuite
+   sudo ./svc.sh start
+
+.. [1]
+   https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
+
+.. [2]
+   https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
diff --git a/run_tests.sh b/run_tests.sh
new file mode 100644
index 00000000..c39788ac
--- /dev/null
+++ b/run_tests.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/bash
+
+# Expects the path to tests folder as the first argument
+if [ $# -eq 0 ]; then
+    echo "No arguments provided. Please provide the path to the tests folder."
+    exit 1
+fi
+
+# Install pytest-html for generating reports
+pip install pytest-html
+
+# Set the path to the reports folder
+REPORTS_DIR="/opt/reports"
+
+# Keep track of failures
+STATUS=0
+
+# If xtrack on Pyopencl context, run tests one by one, otherwise run normally
+if [[ $XOBJECTS_TEST_CONTEXTS =~ "ContextPyopencl" ]] && [[ $* =~ "xsuite/xtrack" ]]; then
+  # Run tests one by one
+  pip install pytest-html-merger
+
+  for test_file in "$@"/test_*; do
+      TEST_NAME=$(basename "$test_file" .py)  # strip path and extension
+      echo "Running test $TEST_NAME..."
+      pytest \
+        --color=yes \
+        --verbose \
+        --html="$REPORTS_DIR/report-$TEST_NAME.html" --self-contained-html \
+        "$test_file" || STATUS=1
+  done
+
+  echo "Generating report..."
+  pytest_html_merger -i "$REPORTS_DIR" -o "$REPORTS_DIR/report.html"
+else
+   # Run tests normally if no Pyopencl context
+  pytest \
+    --color=yes \
+    --verbose \
+    --html="$REPORTS_DIR/report.html" --self-contained-html \
+    "$@" || STATUS=1
+fi
+
+exit $STATUS