Repository: btschwertfeger/python-cmethods (language: python)

Issue:
KeyError when Quantile* Mapping and group != None
**Describe the bug**
When using the `adjust_3d` method for adjusting distribution-based bias correction techniques like the quantile mapping or quantile delta mapping and using a group like `time.month` - an error occurs since the grouping creates Lists inside instead of xarray.datasets. 


Discussion / hints:

Fix patch:
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index dd84ea7..9b77ea7 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -1,10 +1,9 @@
 ---
 name: Bug report
 about: Create a report to help us improve
-title: ''
-labels: ''
-assignees: ''
-
+title: ""
+labels: ""
+assignees: ""
 ---
 
 **Describe the bug**
@@ -12,6 +11,7 @@ A clear and concise description of what the bug is.
 
 **To Reproduce**
 Steps to reproduce the behavior:
+
 1. Go to '...'
 2. Click on '....'
 3. Scroll down to '....'
@@ -24,15 +24,17 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.
 
 **Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
+
+- OS: [e.g. iOS]
+- Browser [e.g. chrome, safari]
+- Version [e.g. 22]
 
 **Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
+
+- Device: [e.g. iPhone6]
+- OS: [e.g. iOS8.1]
+- Browser [e.g. stock browser, safari]
+- Version [e.g. 22]
 
 **Additional context**
 Add any other context about the problem here.
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
index bbcbbe7..2bc5d5f 100644
--- a/.github/ISSUE_TEMPLATE/feature_request.md
+++ b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -1,10 +1,9 @@
 ---
 name: Feature request
 about: Suggest an idea for this project
-title: ''
-labels: ''
-assignees: ''
-
+title: ""
+labels: ""
+assignees: ""
 ---
 
 **Is your feature request related to a problem? Please describe.**
diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml
index 3c77936..699f462 100644
--- a/.github/workflows/codecov.yml
+++ b/.github/workflows/codecov.yml
@@ -20,7 +20,7 @@ jobs:
       fail-fast: false
       max-parallel: 1
       matrix:
-        os: [ubuntu-latest] #, macos-latest, windows-latest]
+        os: [ubuntu-latest]
         python-version: ["3.11"]
 
     env:
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
index 38c4056..fcb5efd 100644
--- a/.github/workflows/codeql.yml
+++ b/.github/workflows/codeql.yml
@@ -74,4 +74,4 @@ jobs:
       - name: Perform CodeQL Analysis
         uses: github/codeql-action/analyze@v2
         with:
-          category: "/language:${{matrix.language}}"
+          category: "/language:${{ matrix.language }}"
diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml
new file mode 100644
index 0000000..879585d
--- /dev/null
+++ b/.github/workflows/pre-commit.yml
@@ -0,0 +1,20 @@
+name: Pre-Commit
+
+on:
+  push:
+    branches: ["*", "!master"]
+  pull_request:
+    branches: ["*"]
+
+concurrency:
+  group: pre-commit-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  pre-commit:
+    name: Pre-Commit
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - uses: pre-commit/action@v3.0.0
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
index 66a6fe3..9aad40a 100644
--- a/.github/workflows/python-package.yml
+++ b/.github/workflows/python-package.yml
@@ -14,13 +14,22 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  pre-commit:
+    name: Pre-Commit
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - uses: pre-commit/action@v3.0.0
+
   build:
+    needs: [pre-commit]
+    name: Build
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         python-version: ["3.8", "3.9", "3.10", "3.11"]
-
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
diff --git a/.gitignore b/.gitignore
index f0054d6..0480b01 100644
--- a/.gitignore
+++ b/.gitignore
@@ -136,4 +136,4 @@ del*.py
 *.zip
 
 *.egg-info/
-conda.stuff/
\ No newline at end of file
+conda.stuff/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 9819f91..930ddc6 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,24 +1,45 @@
 repos:
--   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v2.3.0
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
     hooks:
-    -   id: check-yaml
-    -   id: end-of-file-fixer
-    -   id: trailing-whitespace
-    -   id: check-added-large-files
--   repo: https://github.com/psf/black
-    rev: 22.10.0
+      # all available hooks can be found here: https://github.com/pre-commit/pre-commit-hooks/blob/main/.pre-commit-hooks.yaml
+      - id: check-yaml
+      - id: check-ast
+      - id: check-docstring-first
+      - id: check-case-conflict
+      - id: check-merge-conflict
+      - id: check-added-large-files
+        args: ["--maxkb=500"]
+      - id: trailing-whitespace
+      - id: fix-byte-order-marker
+      - id: fix-encoding-pragma
+      - id: requirements-txt-fixer
+      - id: end-of-file-fixer
+      - id: check-executables-have-shebangs
+      - id: detect-private-key
+  - repo: https://github.com/psf/black
+    rev: 23.1.0
     hooks:
-    -   id: black
-- repo: https://github.com/PyCQA/isort
-  rev: 5.12.0
-  hooks:
-  - id: isort
-    args: ["--profile", "black"] # solves conflicts between black and isort
--   repo: https://github.com/pycqa/flake8
-    rev: '6.0.0'
+      - id: black
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.12.0
     hooks:
-    -   id: flake8
-        # exclude: ^examples/
-        # additional_dependencies: [flake8-docstrings]
+      - id: isort
+        args: ["--profile", "black"] # solves conflicts between black and isort
+  - repo: https://github.com/pycqa/flake8
+    rev: 6.0.0
+    hooks:
+      - id: flake8
         args: ["--select=E9,F63,F7,F82", "--show-source", "--statistics"]
+  - repo: https://github.com/pycqa/pylint
+    rev: v2.17.0
+    hooks:
+      - id: pylint
+        name: pylint
+        types: [python]
+        exclude: ^examples/|^tests/|^setup.py$
+        args: ["--rcfile=.pylintrc"]
+  - repo: https://github.com/pre-commit/mirrors-prettier
+    rev: v2.7.1
+    hooks:
+      - id: prettier
diff --git a/.pylintrc b/.pylintrc
index 8ec0025..7adbaaa 100644
--- a/.pylintrc
+++ b/.pylintrc
@@ -312,8 +312,8 @@ min-public-methods=2
 [EXCEPTIONS]
 
 # Exceptions that will emit a warning when caught.
-overgeneral-exceptions=BaseException,
-                       Exception
+overgeneral-exceptions=builtin.BaseException,
+                       builtin.Exception
 
 
 [FORMAT]
@@ -423,7 +423,10 @@ disable=raw-checker-failed,
         use-symbolic-message-instead,
         anomalous-backslash-in-string,
         multiple-statements,
-        line-too-long
+        line-too-long,
+        import-error,
+        consider-using-with
+
 
 # Enable the message, report, category or checker with the given id(s). You can
 # either give multiple identifier separated by comma (,) or put this option
diff --git a/README.md b/README.md
index 0850519..5fb8737 100644
--- a/README.md
+++ b/README.md
@@ -17,8 +17,7 @@
 
 This Python module contains a collection of different scale- and distribution-based bias adjustment techniques for climatic research (see `/examples/examples.ipynb` for help).
 
-
-### 📍 For the application of bias corrections on *lage data sets* it is recomanded to use the C++ tool [BiasAdjustCXX](https://github.com/btschwertfeger/Bias-Adjustment-Cpp) since bias corrections are complex statistical transformation which are very slow in Python compared to the C++ implementation.
+### 📍 For the application of bias corrections on _lage data sets_ it is recomanded to use the C++ tool [BiasAdjustCXX](https://github.com/btschwertfeger/Bias-Adjustment-Cpp) since bias corrections are complex statistical transformation which are very slow in Python compared to the C++ implementation.
 
 ---
 
diff --git a/cmethods/CMethods.py b/cmethods/CMethods.py
index 875bd44..401a17d 100644
--- a/cmethods/CMethods.py
+++ b/cmethods/CMethods.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
-# -*- coding:utf-8 -*-
-# Copyright (C) 2023 Benjamin Thomas Schwertfegerr
+# -*- coding: utf-8 -*-
+# Copyright (C) 2023 Benjamin Thomas Schwertfeger
 # Github: https://github.com/btschwertfeger
 #
 
diff --git a/cmethods/__init__.py b/cmethods/__init__.py
index 512ec51..57b8b04 100644
--- a/cmethods/__init__.py
+++ b/cmethods/__init__.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# -*- coding:utf-8 -*-
-# Copyright (C) 2023 Benjamin Thomas Schwertfegerr
+# -*- coding: utf-8 -*-
+# Copyright (C) 2023 Benjamin Thomas Schwertfeger
 # Github: https://github.com/btschwertfeger
 #
diff --git a/examples/do_bias_correction.py b/examples/do_bias_correction.py
index e5ed850..be296f7 100755
--- a/examples/do_bias_correction.py
+++ b/examples/do_bias_correction.py
@@ -1,22 +1,25 @@
-#!/bin/python3
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
 
-__descrption__ = 'Script to adjust climate biases in 3D Climate data'
-__author__ = 'Benjamin Thomas Schwertfeger'
+__descrption__ = "Script to adjust climate biases in 3D Climate data"
+__author__ = "Benjamin Thomas Schwertfeger"
 __copyright__ = __author__
-__email__ = 'development@b-schwertfeger.de'
-__link__ = 'https://b-schwertfeger.de'
-__github__ = 'https://github.com/btschwertfeger/Bias-Adjustment-Python';
+__email__ = "development@b-schwertfeger.de"
+__link__ = "https://b-schwertfeger.de"
+__github__ = "https://github.com/btschwertfeger/Bias-Adjustment-Python"
 
 import argparse
-import logging, sys
+import logging
+import sys
+
 import xarray as xr
 
 from cmethods.CMethods import CMethods
 
 # * ----- L O G G I N G -----
 formatter = logging.Formatter(
-    fmt='%(asctime)s %(module)s,line: %(lineno)d %(levelname)8s | %(message)s',
-    datefmt='%Y-%m-%d %H:%M:%S'
+    fmt="%(asctime)s %(module)s,line: %(lineno)d %(levelname)8s | %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
 )
 
 log = logging.getLogger()
@@ -26,76 +29,121 @@
 logging.getLogger().addHandler(screen_handler)
 
 # * ----- I N P U T - H A N D L I N G -----
-parser = argparse.ArgumentParser(description='Adjust climate data based on bias correction algorithms and magic.')
-parser.add_argument('--obs', '--observation', dest='obs_fpath', type=str, help='Observation dataset')
-parser.add_argument('--contr', '--control', dest='contr_fpath', type=str, help='Control dataset')
-parser.add_argument('--scen', '--scenario', dest='scen_fpath', type=str, help='Scenario dataset (data to adjust)')
+parser = argparse.ArgumentParser(
+    description="Adjust climate data based on bias correction algorithms and magic."
+)
+parser.add_argument(
+    "--obs", "--observation", dest="obs_fpath", type=str, help="Observation dataset"
+)
+parser.add_argument(
+    "--contr", "--control", dest="contr_fpath", type=str, help="Control dataset"
+)
+parser.add_argument(
+    "--scen",
+    "--scenario",
+    dest="scen_fpath",
+    type=str,
+    help="Scenario dataset (data to adjust)",
+)
 
-parser.add_argument('-m', '--method', dest='method', type=str, help='Correction method')
-parser.add_argument('-v', '--variable', dest='var', type=str, default='tas', help='Variable to adjust')
-parser.add_argument('-u', '--unit', dest='unit', type=str, default='°C', help='Unit of the varible')
+parser.add_argument("-m", "--method", dest="method", type=str, help="Correction method")
+parser.add_argument(
+    "-v", "--variable", dest="var", type=str, default="tas", help="Variable to adjust"
+)
+parser.add_argument(
+    "-u", "--unit", dest="unit", type=str, default="°C", help="Unit of the varible"
+)
 
-parser.add_argument('-g', '--group', dest='group', type=str, default=None, help='Value grouping, default: time, (options: time.month, time.dayofyear, time.year')
-parser.add_argument('-k', '--kind', dest='kind', type=str, default='+', help='+ or *, default: +')
-parser.add_argument('-n', '--nquantiles', dest='n_quantiles', type=int, default=100, help='Nr. of Quantiles to use')
+parser.add_argument(
+    "-g",
+    "--group",
+    dest="group",
+    type=str,
+    default=None,
+    help="Value grouping, default: time, (options: time.month, time.dayofyear, time.year",
+)
+parser.add_argument(
+    "-k", "--kind", dest="kind", type=str, default="+", help="+ or *, default: +"
+)
+parser.add_argument(
+    "-n",
+    "--nquantiles",
+    dest="n_quantiles",
+    type=int,
+    default=100,
+    help="Nr. of Quantiles to use",
+)
 
-parser.add_argument('-p', '--processes', dest='p', type=int, default=1, help='Multiprocessing with n processes, default: 1')
+parser.add_argument(
+    "-p",
+    "--processes",
+    dest="p",
+    type=int,
+    default=1,
+    help="Multiprocessing with n processes, default: 1",
+)
 params = vars(parser.parse_args())
 
-obs_fpath = params['obs_fpath']
-contr_fpath = params['contr_fpath']
-scen_fpath = params['scen_fpath']
+obs_fpath = params["obs_fpath"]
+contr_fpath = params["contr_fpath"]
+scen_fpath = params["scen_fpath"]
+
+method = params["method"]
+var = params["var"]
+unit = params["unit"]
+group = params["group"]
+kind = params["kind"]
+n_quantiles = params["n_quantiles"]
+n_jobs = params["p"]
 
-method = params['method']
-var = params['var']
-unit = params['unit']
-group = params['group']
-kind = params['kind']
-n_quantiles = params['n_quantiles']
-n_jobs = params['p']
 
 # * ----- ----- -----M A I N ----- ----- -----
 def main() -> None:
     cm = CMethods()
 
-    if method not in cm.get_available_methods(): raise ValueError(f'Unknown method {method}. Available methods: {cm.get_available_methods()}')
+    if method not in cm.get_available_methods():
+        raise ValueError(
+            f"Unknown method {method}. Available methods: {cm.get_available_methods()}"
+        )
 
     ds_obs = xr.open_dataset(obs_fpath)[var]
     ds_simh = xr.open_dataset(contr_fpath)[var]
     ds_simp = xr.open_dataset(scen_fpath)[var]
-    log.info('Data Loaded')
+    log.info("Data Loaded")
 
-    ds_obs.attrs['unit'] = unit
-    ds_simh.attrs['unit'] = unit
-    ds_simp.attrs['unit'] = unit
+    ds_obs.attrs["unit"] = unit
+    ds_simh.attrs["unit"] = unit
+    ds_simp.attrs["unit"] = unit
 
-    start_date: str = ds_simp['time'][0].dt.strftime('%Y%m%d').values.ravel()[0]
-    end_date: str = ds_simp['time'][-1].dt.strftime('%Y%m%d').values.ravel()[0]
+    start_date: str = ds_simp["time"][0].dt.strftime("%Y%m%d").values.ravel()[0]
+    end_date: str = ds_simp["time"][-1].dt.strftime("%Y%m%d").values.ravel()[0]
 
-    descr1, descr2 = '', ''
+    descr1, descr2 = "", ""
     if method in cm.DISTRIBUTION_METHODS:
-        descr1 = f'_quantiles-{n_quantiles}'
+        descr1 = f"_quantiles-{n_quantiles}"
 
     # ----- Adjustment -----
-    log.info(f'Starting {method} adjustment')
+    log.info(f"Starting {method} adjustment")
     result = cm.adjust_3d(
-        method = method,
-        obs = ds_obs,
-        simh = ds_simh,
-        simp = ds_simp,
-        n_quantiles = n_quantiles,
-        kind = kind,
-        group = group,
-        n_jobs = n_jobs
+        method=method,
+        obs=ds_obs,
+        simh=ds_simh,
+        simp=ds_simp,
+        n_quantiles=n_quantiles,
+        kind=kind,
+        group=group,
+        n_jobs=n_jobs,
     )
-    log.info('Saving now')
+    log.info("Saving now")
     result.name = var
-    result['time'] = ds_simp['time']
-    result.to_netcdf(f'{method}_result_var-{var}{descr1}_kind-{kind}_group-{group}{descr2}_{start_date}_{end_date}.nc')
-    log.info('Done')
+    result["time"] = ds_simp["time"]
+    result.to_netcdf(
+        f"{method}_result_var-{var}{descr1}_kind-{kind}_group-{group}{descr2}_{start_date}_{end_date}.nc"
+    )
+    log.info("Done")
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()
 
 
diff --git a/requirements.txt b/requirements.txt
index 894c297..5eb413c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,4 @@
-xarray>=2022.11.0
 netCDF4>=1.6.1
 numpy
-tqdm
\ No newline at end of file
+tqdm
+xarray>=2022.11.0
diff --git a/run_pylint.sh b/run_pylint.sh
deleted file mode 100755
index 57830f8..0000000
--- a/run_pylint.sh
+++ /dev/null
@@ -1,7 +0,0 @@
-#!/bin/bash
-
-# remove trailing whitespaces
-find cmethods -type f -name '*.py' -exec sed -i '' 's/[[:space:]]*$//' {} \+
-
-# run pylint
-pylint --rcfile=.pylintrc cmethods 
\ No newline at end of file
diff --git a/setup.py b/setup.py
index b975700..c7e1aa4 100644
--- a/setup.py
+++ b/setup.py
@@ -1,6 +1,7 @@
-#!/usr/bin/python3
-# -*- coding:utf-8 -*-
-# run via  python3 setup.py upload
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright (C) 2023 Benjamin Thomas Schwertfeger
+# Github: https://github.com/btschwertfeger
 
 import io
 import os
@@ -9,7 +10,6 @@
 
 from setuptools import Command, find_packages, setup
 
-# Package meta-data.
 NAME = "python-cmethods"
 DESCRIPTION = (
     "Collection of bias adjustment procedures for multidimensional climate data"
@@ -18,7 +18,7 @@
 EMAIL = "development@b-schwertfeger.de"
 AUTHOR = "Benjamin Thomas Schwertfeger"
 REQUIRES_PYTHON = ">=3.8.0"
-VERSION = "0.6.1"
+VERSION = "0.6.2"
 
 # What packages are required for this module to be executed?
 REQUIRED = [


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
index 512ec51..fdfa67e 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# -*- coding:utf-8 -*-
+# -*- coding: utf-8 -*-
 # Copyright (C) 2023 Benjamin Thomas Schwertfegerr
 # Github: https://github.com/btschwertfeger
 #
diff --git a/tests/requirements.txt b/tests/requirements.txt
index aeb27d2..eed36be 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,3 +1,3 @@
-scikit-learn
 numpy
+scikit-learn
 xarray
diff --git a/tests/test_methods.py b/tests/test_methods.py
index c313c7e..9e0286c 100644
--- a/tests/test_methods.py
+++ b/tests/test_methods.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# -*- coding:utf-8 -*-
+# -*- coding: utf-8 -*-
 # Copyright (C) 2023 Benjamin Thomas Schwertfegerr
 # Github: https://github.com/btschwertfeger
 #