Repository: CPMpy/cpmpy
Pull request #668: Skip non-pysat-specific tests on missing pblib
Author: login: ThomSerg
Created: 2025-05-05T16:14:49Z | Merged: 2025-05-07T08:13:21Z

Continuation of #652. Adds a set of useful decorators to aid in applying other decorators to individual tests or to entire test cases. For example, skipping a test if an expected exception occurs. 

As an application, the decorator `@skip_on_missing_pblib()` is proposed (internally makes use of the other decorators). It allows skipping non-pysat-specific tests that fail due to missing optional `pblib` dependency. The test don't actually get skipped, but their failure gets captured and silenced as if it were skipped. This simple decorator prevents a lot of logic that would otherwise be required to skip these tests before (instead of after) running them.

Requires #652 to be merged first.


Diff:
diff --git a/tests/test_constraints.py b/tests/test_constraints.py
index ef3684591..aadcdf09a 100644
--- a/tests/test_constraints.py
+++ b/tests/test_constraints.py
@@ -8,6 +8,8 @@
 
 import pytest
 
+from utils import skip_on_missing_pblib
+
 # CHANGE THIS if you want test a different solver
 #   make sure that `SolverLookup.get(solver)` works
 # also add exclusions to the 3 EXCLUDE_* below as needed
@@ -272,6 +274,7 @@ def verify(cons):
 
 
 @pytest.mark.parametrize(("solver","constraint"),list(_generate_inputs(bool_exprs)), ids=str)
+@skip_on_missing_pblib(skip_on_exception_only=True)
 def test_bool_constraints(solver, constraint):
     """
         Tests boolean constraint by posting it to the solver and checking the value after solve.
@@ -286,6 +289,7 @@ def test_bool_constraints(solver, constraint):
 
 
 @pytest.mark.parametrize(("solver","constraint"), list(_generate_inputs(comp_constraints)),  ids=str)
+@skip_on_missing_pblib(skip_on_exception_only=True)
 def test_comparison_constraints(solver, constraint):
     """
         Tests comparison constraint by posting it to the solver and checking the value after solve.
@@ -300,6 +304,7 @@ def test_comparison_constraints(solver, constraint):
 
 
 @pytest.mark.parametrize(("solver","constraint"), list(_generate_inputs(reify_imply_exprs)),  ids=str)
+@skip_on_missing_pblib(skip_on_exception_only=True)
 def test_reify_imply_constraints(solver, constraint):
     """
         Tests boolean expression by posting it to solver and checking the value after solve.
diff --git a/tests/test_globalconstraints.py b/tests/test_globalconstraints.py
index aaf4efa49..f3846165f 100644
--- a/tests/test_globalconstraints.py
+++ b/tests/test_globalconstraints.py
@@ -9,7 +9,10 @@
 from cpmpy.expressions.utils import STAR
 from cpmpy.solvers import CPM_minizinc
 
+from utils import skip_on_missing_pblib
 
+
+@skip_on_missing_pblib(skip_on_exception_only=True)
 class TestGlobal(unittest.TestCase):
     def test_alldifferent(self):
         """Test all different constraint with a set of
diff --git a/tests/test_pysat_cardinality.py b/tests/test_pysat_cardinality.py
index 04b20672e..c3dc2f61d 100644
--- a/tests/test_pysat_cardinality.py
+++ b/tests/test_pysat_cardinality.py
@@ -6,6 +6,7 @@
 from cpmpy.solvers.pysat import CPM_pysat
 from cpmpy.transformations.linearize import only_positive_coefficients
 
+from utils import skip_on_missing_pblib
 
 SOLVER = "pysat"
 
@@ -134,6 +135,7 @@ def test_pysat_different(self):
         self.assertTrue(ps.solve())
         self.assertGreaterEqual(sum(self.bvs.value()), 2)
 
+    @skip_on_missing_pblib()
     def test_pysat_card_implied(self):
         b = cp.boolvar()
         x = cp.boolvar(shape=5)
diff --git a/tests/test_pysat_wsum.py b/tests/test_pysat_wsum.py
index bbd8451f1..1de62dde3 100644
--- a/tests/test_pysat_wsum.py
+++ b/tests/test_pysat_wsum.py
@@ -5,9 +5,10 @@
 from cpmpy.solvers.pysat import CPM_pysat
 
 import importlib # can check for modules *without* importing them
+pysat_available = CPM_pysat.supported()
 pblib_available = importlib.util.find_spec("pypblib") is not None
 
-@pytest.mark.skipif(pblib_available, reason="`pypblib` is installed")
+@pytest.mark.skipif(not (pysat_available and not pblib_available), reason="`pysat` is not installed" if not pysat_available else "`pypblib` is installed")
 def test_pypblib_error():
     # NOTE if you want to run this but pypblib is already installed, run `pip uninstall pypblib && pip install -e .[pysat]`
     unittest.TestCase().assertRaises(
@@ -18,7 +19,7 @@ def test_pypblib_error():
     # this one should still work without `pypblib`
     assert CPM_pysat(cp.Model(1*cp.boolvar() + 1 * cp.boolvar() + 1 * cp.boolvar() <= 2)).solve()
 
-@pytest.mark.skipif(not pblib_available, reason="pypblib not installed")
+@pytest.mark.skipif(not (pysat_available and pblib_available), reason="`pysat` is not installed" if not pysat_available else "`pypblib` not installed")
 class TestEncodePseudoBooleanConstraint(unittest.TestCase):
     def setUp(self):
         self.bv = boolvar(shape=3)
diff --git a/tests/test_solvers.py b/tests/test_solvers.py
index 0db8ef331..bb470a87e 100644
--- a/tests/test_solvers.py
+++ b/tests/test_solvers.py
@@ -1,3 +1,4 @@
+import importlib
 import unittest
 import tempfile
 import pytest
@@ -15,6 +16,11 @@
 from cpmpy import SolverLookup
 from cpmpy.exceptions import MinizincNameException, NotSupportedError
 
+from utils import skip_on_missing_pblib
+
+pysat_available = CPM_pysat.supported()
+pblib_available = importlib.util.find_spec("pypblib") is not None
+
 class TestSolvers(unittest.TestCase):
 
     # should move this test elsewhere later
@@ -313,8 +319,8 @@ def test_pysat(self):
         self.assertFalse(ps2.solve(assumptions=[mayo]+[v for v in inds]))
         self.assertEqual(ps2.get_core(), [mayo,inds[6],inds[9]])
 
-    @pytest.mark.skipif(not CPM_pysat.supported(),
-                        reason="PySAT not installed")
+    @pytest.mark.skipif(not (pysat_available and pblib_available), reason="`pysat` is not installed" if not pysat_available else "`pypblib` not installed")
+    @skip_on_missing_pblib()
     def test_pysat_card(self):
         b = cp.boolvar()
         x = cp.boolvar(shape=5)
diff --git a/tests/utils.py b/tests/utils.py
new file mode 100644
index 000000000..35afd0c2c
--- /dev/null
+++ b/tests/utils.py
@@ -0,0 +1,105 @@
+import importlib
+import pytest
+from functools import wraps
+import cpmpy as cp
+
+# ---------------------------------------------------------------------------- #
+#                              Generic Decorators                              #
+# ---------------------------------------------------------------------------- #
+
+"""
+These should probably not be used on their own, 
+but rather as building blocks for the more "specific" decorators below.
+"""
+
+def skip_on_exception(exc_type, message_contains=None, skip_message=None):
+    """
+    Skip test when expected failure occurs.
+    """
+
+    def decorator(func):
+        """
+        The actual decorator that gets placed around the function.
+        """
+        @wraps(func)
+        def wrapper(*args, **kwargs):
+            """
+            The function that gets called instead of the decorated function.
+            """
+            try:
+                # Try calling the decorated function
+                return func(*args, **kwargs)
+            except exc_type as e:
+                # Check if expected exception
+                if message_contains is None or message_contains in str(e):
+                    msg = skip_message or f"Skipped due to {exc_type.__name__}: {e}"
+                    pytest.skip(msg) # expected -> skip test
+                raise  # Re-raise if not the expected exception
+        return wrapper
+    return decorator
+
+
+def apply_decorator_to_tests(decorator):
+    """
+    Decorator wrapper for unittest.TestCase classes.
+    Applies `decorator` to all methods with a name starting with `test_`.
+    """
+    def class_decorator(cls):
+        for attr_name in dir(cls):
+            if attr_name.startswith("test_"):
+                attr = getattr(cls, attr_name)
+                if callable(attr):
+                    setattr(cls, attr_name, decorator(attr))
+        return cls
+    return class_decorator
+
+
+def smart_decorator(method_decorator):
+    """
+    Wraps a method decorator so it can be applied to either:
+    - a function/method: applies the decorator directly
+    - a class: applies the decorator to all test_* methods
+    """
+    def wrapper(obj):
+        if isinstance(obj, type):
+            return apply_decorator_to_tests(method_decorator)(obj)
+        elif callable(obj):
+            return method_decorator(obj)
+        else:
+            raise TypeError("smart_decorator can only be used on classes or callables")
+    return wrapper
+
+
+# ---------------------------------------------------------------------------- #
+#                              Specific Decorators                             #
+# ---------------------------------------------------------------------------- #
+
+from cpmpy.solvers.pysat import CPM_pysat
+pblib_available = importlib.util.find_spec("pypblib") is not None
+
+def skip_on_missing_pblib(skip_on_exception_only:bool=False):
+    """
+    Skips the decorated test when the optional `pblib` dependency is not available on the current system.
+
+    Arguments:
+        skip_on_exception_only (bool): If set to `True`, still run the test but ignore any exception related to missing `pblib`.
+                                        If exception occurs, test gets reported as being skipped.
+                                       If set to `False`, test doesn't get run (even if test does not rely on `pblib`) 
+
+    Notes:
+        `@skip_on_missing_pblib()` should be used for test which we know require `pblib`. 
+            These tests then never get run, reducing runtime.
+        `@skip_on_missing_pblib(skip_on_exception_only=True)` should be used when we're not sure if it requires 
+            `pblib` but would like to ignore any exception related to missing `pblib` dependency.
+    """
+
+    if not skip_on_exception_only:
+        return pytest.mark.skipif(not pblib_available, reason="`pypblib` not installed")
+    
+    return smart_decorator(
+        skip_on_exception(
+            ImportError,
+            message_contains="PB constraint",
+            skip_message="`pypblib` not installed"
+        )
+    )


Reviews:
user: login: IgnaceBleukx
body: I'm a bit confused, is this PR undoing some changes from #652? 
It seems they're trying to solve the same thing, but differently.
If we have a `skip_on_missing_pblib` maybe it also makes sense to have a `skip_on_missing_pysat_or_pblib`? Now this PR tries to clean things up, but leaves some room for improvement I feel
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-06T08:47:51Z

Review comments:
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+from functools import wraps
+
+def skip_on_exception(exc_type, message_contains=None, skip_message=None):
+    """
+    Skip test when expected failure occurs.
+    """
+    def decorator(func):
+        @wraps(func)
+        def wrapper(*args, **kwargs):
+            try:
+                return func(*args, **kwargs)
+            except exc_type as e:
+                if message_contains is None or message_contains in str(e):
+                    msg = skip_message or f"Skipped due to {exc_type.__name__}: {e}"
+                    pytest.skip(msg)
+                raise  # Re-raise if the message doesn't match
+        return wrapper
+    return decorator
+
+
+def apply_decorator_to_tests(decorator):
+    """
+    Decorator wrapper for unittest.TestCase classes.
+    Applies `decorator` to all methods with a name starting with `test_`.
+    """
+    def class_decorator(cls):
+        for attr_name in dir(cls):
+            if attr_name.startswith("test_"):
+                attr = getattr(cls, attr_name)
+                if callable(attr):
+                    setattr(cls, attr_name, decorator(attr))
+        return cls
+    return class_decorator
+
+
+def smart_decorator(method_decorator):
+    """
+    Wraps a method decorator so it can be applied to either:
+    - a function/method: applies the decorator directly
+    - a class: applies the decorator to all test_* methods
+    """
+    def wrapper(obj):
+        if isinstance(obj, type):
path: tests/utils.py
user: login: IgnaceBleukx
body: This is a bit criptic and sneaky, can you add a comment to explain what this does exactly?
created_at: 2025-05-06T08:42:43Z
author_association: COLLABORATOR
diff_hunk: @@ -134,6 +136,7 @@ def test_pysat_different(self):
         self.assertTrue(ps.solve())
         self.assertGreaterEqual(sum(self.bvs.value()), 2)
 
+    @pytest.mark.skipif(not pblib_available, reason="`pypblib` not installed")
path: tests/test_pysat_cardinality.py
user: login: IgnaceBleukx
body: Shouldn't this one also be "skip_on_missing_pb()"?
created_at: 2025-05-06T08:44:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+from functools import wraps
+
+def skip_on_exception(exc_type, message_contains=None, skip_message=None):
+    """
+    Skip test when expected failure occurs.
+    """
+    def decorator(func):
+        @wraps(func)
+        def wrapper(*args, **kwargs):
+            try:
+                return func(*args, **kwargs)
+            except exc_type as e:
+                if message_contains is None or message_contains in str(e):
+                    msg = skip_message or f"Skipped due to {exc_type.__name__}: {e}"
+                    pytest.skip(msg)
+                raise  # Re-raise if the message doesn't match
+        return wrapper
+    return decorator
+
+
+def apply_decorator_to_tests(decorator):
+    """
+    Decorator wrapper for unittest.TestCase classes.
+    Applies `decorator` to all methods with a name starting with `test_`.
+    """
+    def class_decorator(cls):
+        for attr_name in dir(cls):
+            if attr_name.startswith("test_"):
+                attr = getattr(cls, attr_name)
+                if callable(attr):
+                    setattr(cls, attr_name, decorator(attr))
+        return cls
+    return class_decorator
+
+
+def smart_decorator(method_decorator):
+    """
+    Wraps a method decorator so it can be applied to either:
+    - a function/method: applies the decorator directly
+    - a class: applies the decorator to all test_* methods
+    """
+    def wrapper(obj):
+        if isinstance(obj, type):
+            return apply_decorator_to_tests(method_decorator)(obj)
+        elif callable(obj):
+            return method_decorator(obj)
+        else:
+            raise TypeError("smart_decorator can only be used on classes or callables")
+    return wrapper
+
+def skip_on_missing_pblib():
path: tests/utils.py
user: login: IgnaceBleukx
body: Add a docstring here saying what it means
created_at: 2025-05-06T08:46:38Z
author_association: COLLABORATOR

PR comments:
user: login: ThomSerg
created_at: 2025-05-06T09:54:02Z
body: Both pull requests (this one and #652) indeed try to achieve similar goals, but they tackle different subproblems. So it doesn't clean up #652 (but it could, see the end).

In the case of #652, these are tests specifically tailored to PySAT where we know beforehand a missing pblib will cause an exception (due to the presence of PB constraints). By decorating with
```python
@pytest.mark.skipif(not (pysat_available and not pblib_available), reason="`pysat` is not installed" if not pysat_available else "`pypblib` is installed")
```
these tests don't get run when either pysat of pblib is missing.

In this pull request, we're solving the issue of more generic tests with models that after transformation result in PB constraints. When these tests get run on PySAT and pblib is missing, again an exception will occur. Unlike the above setting, here it is more tricky to know if some generic model anywhere in our test set will lead to PB constraints. So as to not clutter the test code too much, I created the `skip_on_missing_pblib` decorator. It simply wraps a try..catch around the testcase and if it raises an error due to missing pblib, we simply ignore its result. So the biggest difference is that the test still gets run (but their failure is ignored). 

I did some experiments, and it is possible to integrate the decorator of #652 into `skip_on_missing_pblib`, we'll just need some optional argument to describe which behavior of the two you want, i.e. "always skip on missing pblib" or "skip only if pblib causes an issue". E.g:

```python
@skip_on_missing_pblib()
def test_we_know_needs_pblib():
    ...
@skip_on_missing_pblib(defer=True)
def generic_test_that_could_fail_on_pblib():
    ...
``` 
Maybe the `defer` argument should be more descriptive like `skip_on_exception_only`

We can't just manually put `@skip_on_missing_pblib()` on all the test, since a lot of them are generated dynamically.

With regards to `skip_on_missing_pysat_or_pblib`, it would only be used on the PySAT-specific test, so not sure if we would benefit that much from abstracting it.
author_association: COLLABORATOR
user: login: ThomSerg
created_at: 2025-05-06T11:10:10Z
body: With the current implementation #652 is fully integrated in this pull request and would thus no longer be required.

I looked into `skip_on_missing_pysat_or_pblib`, but is seems like this is currently not needed. Instead it is more like:
```python
@skip_on_missing_pysat() # <- does not exist yet
class TestCardinality(unittest.TestCase):
    ...
    @skip_on_missing_pblib()
    def test_pysat_card_implied(self):
        ...
```

So the checks occur at different levels.

---

Branch [skip_on_missing_solver](https://github.com/CPMpy/cpmpy/blob/skip_on_missing_solver/tests/utils.py) contains a possible followup:
```python
@skip_on_missing_solver("<solver_name>")
```
to replace
```python
@pytest.mark.skipif(not CPM_pysat.supported(),
                    reason="PySAT not installed")
```

e.g. `@skip_on_missing_solver("pysat")` for the above missing `@skip_on_missing_pysat()`.

author_association: COLLABORATOR
user: login: hbierlee
created_at: 2025-05-06T16:01:36Z
body: I'm happy with this decorator, it will allow us to run more tests with solvers where we are not sure whether they support the model or not. With the generic version, we can also remove some of the `if solver (..) : continue` in the future.

Unfortunate we need so much engineering just because of supporting a solver with- and without optional dependency.
author_association: CONTRIBUTOR
user: login: IgnaceBleukx
created_at: 2025-05-07T07:49:52Z
body: Ok I see -- so yes this one is fine to merge for me; let's revisit the engineering challenge for making the decorator more generic in the future.
author_association: COLLABORATOR