Repository: XENONnT/straxen (language: python)

Issue:
Stop supporting/testing with python 3.6
Similar to https://github.com/AxFoundation/strax/issues/503, the [straxen requirements file for testing](https://github.com/XENONnT/straxen/blob/master/extra_requirements/requirements-tests.txt)


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/code_style.yml b/.github/workflows/code_style.yml
index 9d1473c5c..d7c3862b5 100644
--- a/.github/workflows/code_style.yml
+++ b/.github/workflows/code_style.yml
@@ -11,7 +11,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@master
         with:
-          python-version: 3.6
+          python-version: 3.8
       - name: Change __all__ exports for pyflake
         run: |
              bash .github/scripts/pre_pyflakes.sh
diff --git a/.github/workflows/contexts.yml b/.github/workflows/contexts.yml
index d83de2bea..75839d396 100644
--- a/.github/workflows/contexts.yml
+++ b/.github/workflows/contexts.yml
@@ -17,7 +17,7 @@ jobs:
     - name: Setup python
       uses: actions/setup-python@v2
       with:
-        python-version: '3.6'
+        python-version: '3.8'
     - name: Checkout repo
       uses: actions/checkout@v2
     - name: Install python dependencies
diff --git a/.github/workflows/coveralls.yml b/.github/workflows/coveralls.yml
deleted file mode 100644
index 8d1b880ce..000000000
--- a/.github/workflows/coveralls.yml
+++ /dev/null
@@ -1,65 +0,0 @@
-on:
-  workflow_dispatch:
-  release:
-    types: [created]
-  pull_request:
-  push:
-    branches:
-      - master
-
-name: Coveralls
-
-jobs:
-
-  build:
-    name: Coveralls
-    runs-on: ubuntu-latest
-    steps:
-
-    - uses: actions/checkout@v1
-    - name: Setup python
-      uses: actions/setup-python@v2 # https://github.com/marketplace/actions/setup-miniconda
-      with:
-        python-version: 3.8
-    - name: Checkout repo
-      uses: actions/checkout@v2
-    - name: Install requirements for tests and latest strax
-      run: |
-        pip install -r extra_requirements/requirements-tests.txt
-        pip install git+https://github.com/AxFoundation/strax.git
-    - name: patch utilix file
-      run: bash .github/scripts/create_readonly_utilix_config.sh
-      env:
-        # Database
-        RUNDB_API_URL: ${{ secrets.RUNDB_API_URL }}
-        RUNDB_API_USER_READONLY: ${{ secrets.RUNDB_API_USER_READONLY }}
-        RUNDB_API_PASSWORD_READONLY: ${{ secrets.RUNDB_API_PASSWORD_READONLY}}
-        PYMONGO_URL: ${{ secrets.PYMONGO_URL }}
-        PYMONGO_USER: ${{ secrets.PYMONGO_USER }}
-        PYMONGO_PASSWORD: ${{ secrets.PYMONGO_PASSWORD }}
-        PYMONGO_DATABASE: ${{ secrets.PYMONGO_DATABASE }}
-        # SCADA
-        SCADA_URL: ${{ secrets.SCADA_URL }}
-        SCADA_VALUE_URL: ${{ secrets.SCADA_VALUE_URL }}
-        SCADA_USER: ${{ secrets.SCADA_USER }}
-        SCADA_LOGIN_URL: ${{ secrets.SCADA_LOGIN_URL }}
-        SCADA_PWD: ${{ secrets.SCADA_PWD }}
-    - name: Create pre-apply function file
-      # In case we do not have database. We need to make a local file for
-      # The pre_apply_function (see #559)
-      env:
-        HAVE_ACCESS_TO_SECTETS: ${{ secrets.RUNDB_API_URL }}
-      if: env.HAVE_ACCESS_TO_SECTETS == null
-      run: |
-        bash .github/scripts/create_pre_apply_function.sh $HOME
-    - name: Coveralls
-      env:
-        NUMBA_DISABLE_JIT: 1
-      run: |
-        coverage run --source=straxen setup.py test -v
-    - name: Upload coverage
-      env:
-        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
-      if: env.COVERALLS_REPO_TOKEN != null
-      run: |
-         coveralls
diff --git a/.github/workflows/pypi_install.yml b/.github/workflows/pypi_install.yml
index 3d8d5dc9c..394dc3ed8 100644
--- a/.github/workflows/pypi_install.yml
+++ b/.github/workflows/pypi_install.yml
@@ -15,7 +15,7 @@ jobs:
       - name: Setup python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.6'
+          python-version: '3.8'
       - name: Checkout repo
         uses: actions/checkout@v2
       - name: Install dependencies
diff --git a/docs/source/setup.rst b/docs/source/setup.rst
index 72215e026..df82cd9a1 100644
--- a/docs/source/setup.rst
+++ b/docs/source/setup.rst
@@ -1,7 +1,7 @@
 Setting up straxen
 ===================
 
-To install straxen locally, follow these steps in a python 3.6, 3.7 or 3.8 environment:
+To install straxen locally, follow these steps in a python 3.7, 3.8 or 3.9 environment:
 
 1. `git clone https://github.com/XENONnT/straxen`
 2. **Optional**. If you are NOT on the UChicago Midway analysis center please follow step 2. in this `wiki note (restricted) <https://xe1t-wiki.lngs.infn.it/doku.php?id=xenon:xenonnt:analysis:guide:straxen_installation>`_.
diff --git a/requirements.txt b/requirements.txt
index f91cf09b3..30613a3bc 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -10,5 +10,4 @@ immutabledict
 numba>=0.50.0
 requests
 commentjson
-matplotlib; python_version>="3.7"
-matplotlib<3.4; python_version=="3.6"
+matplotlib
diff --git a/setup.py b/setup.py
index b9dffec88..c564e5642 100644
--- a/setup.py
+++ b/setup.py
@@ -30,6 +30,7 @@ def open_requirements(path):
                  setup_requires=['pytest-runner'],
                  install_requires=requires,
                  tests_require=requires + tests_requires,
+                 # Not testing py3.6 #616
                  python_requires=">=3.6",
                  extras_require={
                      'docs': doc_requirements,
diff --git a/straxen/misc.py b/straxen/misc.py
index 695905593..6e41ef0be 100644
--- a/straxen/misc.py
+++ b/straxen/misc.py
@@ -9,10 +9,9 @@
 import pytz
 from os import environ as os_environ
 from importlib import import_module
-
+from configparser import NoSectionError
 
 export, __all__ = strax.exporter()
-from configparser import NoSectionError
 
 
 @export


Test patch:
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
index 28f2ac3d5..26c334edc 100644
--- a/.github/workflows/pytest.yml
+++ b/.github/workflows/pytest.yml
@@ -1,4 +1,15 @@
-# Test straxen on each PR
+# Test straxen on each PR.
+# We run three types of tests:
+#  - Pytest -> these are the "normal" tests and should be run for all
+#    python versions
+#  - Coveralls -> this is to see if we are coverering all our lines of
+#    code with our tests. The results get uploaded to
+#    coveralls.io/github/XENONnT/straxen
+#  - pytest_no_database -> we want to make sure we can run the tests even
+#    if we don't have access to our datebase since this will e.g. happen
+#    when someone is pushing a PR from their own fork as we don't
+#    propagate our secrets there.
+
 name: Test package
 
 # Trigger this code when a new release is published
@@ -16,14 +27,25 @@ on:
 
 jobs:
   update:
-    name: Pytest
+    name: "${{ matrix.test }}_py${{ matrix.python-version }}"
     runs-on: ubuntu-latest
     strategy:
-      fail-fast: true
+      fail-fast: True
       matrix:
-        # NB: For python 3.6 and 3.7 we are going to explicitly test without a database
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        python-version: [3.7, 3.8, 3.9]
+        test: ['coveralls', 'pytest', 'pytest_no_database']
+        # Only run coverage / no_database on py3.8
+        exclude:
+          - python-version: 3.7
+            test: coveralls
+          - python-version: 3.9
+            test: coveralls
+          - python-version: 3.7
+            test: pytest_no_database
+          - python-version: 3.9
+            test: pytest_no_database
     steps:
+      # Setup and installation
       - name: Setup python
         uses: actions/setup-python@v2
         with:
@@ -34,12 +56,14 @@ jobs:
         run: |
             pip install -r extra_requirements/requirements-tests.txt
             pip install git+https://github.com/AxFoundation/strax.git
+
+      # Secrets and required files
       - name: patch utilix file
-        # Since we skip this step for python 3.6 and 3.7, we are testing without the database
-        if:  matrix.python-version == 3.8 || matrix.python-version == 3.9
+        # Patch this file if we want to have access to the database
+        if:  matrix.test != 'pytest_no_database'
         run: bash .github/scripts/create_readonly_utilix_config.sh
         env:
-          # Rundb
+          # RunDB
           RUNDB_API_URL: ${{ secrets.RUNDB_API_URL }}
           RUNDB_API_USER_READONLY: ${{ secrets.RUNDB_API_USER_READONLY }}
           RUNDB_API_PASSWORD_READONLY: ${{ secrets.RUNDB_API_PASSWORD_READONLY}}
@@ -55,14 +79,29 @@ jobs:
           SCADA_PWD: ${{ secrets.SCADA_PWD }}
       - name: Create pre-apply function file
         # In case we do not have database. We need to make a local file for
-        # The pre_apply_function (see #559)
+        # The pre_apply_function (see #559).
         env:
           HAVE_ACCESS_TO_SECTETS: ${{ secrets.RUNDB_API_URL }}
-        if: env.HAVE_ACCESS_TO_SECTETS == null || matrix.python-version == 3.6 || matrix.python-version == 3.7
+        if: env.HAVE_ACCESS_TO_SECTETS == null || matrix.test == 'pytest_no_database'
         run: |
           bash .github/scripts/create_pre_apply_function.sh $HOME
+
+      # Run tests
       - name: Test package
+        # This is running a normal test
+        if: matrix.test == 'pytest_no_database' || matrix.test == 'pytest'
         run: |
           python setup.py test -v
+      - name: Coveralls
+        # Make the coverage report and upload
+        if:  matrix.test == 'coveralls'
+        env:
+          NUMBA_DISABLE_JIT: 1
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          coverage run --source=straxen setup.py test -v
+          coveralls --service=github
+
+      # Done
       - name: goodbye
-        run: echo goodbye
+        run: echo "tests done, bye bye"
diff --git a/extra_requirements/requirements-tests.txt b/extra_requirements/requirements-tests.txt
index 22dab7288..3500f8665 100644
--- a/extra_requirements/requirements-tests.txt
+++ b/extra_requirements/requirements-tests.txt
@@ -2,42 +2,27 @@
 blosc==1.10.4                                   # Strax dependency
 boltons==21.0.0
 datashader==0.13.0
-
-dask~=2021.7; python_version>="3.7"
-dask<=2021.2.0; python_version=="3.6"
-
+dask==2021.7.2
 dill==0.3.4                                     # Strax dependency
 coveralls==3.2.0
 commentjson==0.9.0
 coverage==5.5
 flake8==3.9.2
 holoviews==1.14.5
-importlib-metadata==4.6.3; python_version=="3.7"
 ipywidgets==7.6.3
 hypothesis==6.14.5
 jupyter-client==6.1.12                          # for ipywidgets
 npshmex==0.2.1                                  # Strax dependency
 numba==0.53.1                                   # Strax dependency
 numpy==1.19.5
-
-pandas==1.2.5; python_version>="3.7"            # Strax dependency
-pandas==1.1.5; python_version=="3.6"            # Strax dependency
-
+pandas==1.2.5                                   # Strax dependency
 psutil==5.8.0                                   # Strax dependency
 pytest==6.2.4
 pytest-cov==2.12.1
 scikit-learn==0.24.2
-
-scipy==1.7.0; python_version>="3.7"             # Strax dependency
-scipy==1.5.4; python_version=="3.6"             # Strax dependency
-
-tensorflow~=2.5.0; python_version=="3.6"
-tensorflow~=2.4; python_version>="3.7"
-
+scipy==1.7.0                                    # Strax dependency
+tensorflow==2.5.0
 tqdm==4.62.0
-
-xarray==0.19.0; python_version>="3.7"
-xarray==0.16.2; python_version=="3.6"
-
+xarray==0.19.0
 utilix==0.6.1
 zstd==1.5.0.2                                   # Strax dependency