Repository: pytest-dev/pytest-order (language: python)

Issue:
Internal errors while testing pytest-dependency
Hello o/

I've ran into some issues while running tests for https://github.com/RKrahl/pytest-dependency/pull/44. Since their test suite always enables all plugins and there isn't an easy way to disable one for the tests, I've had to temporarily uninstall the plugin to finish testing. I figured I'd let you know that there are some unhandled edge cases somewhere in your plugin.

Using `order-1.0.0`, `platform win32 -- Python 3.8.8, pytest-6.2.5, py-1.10.0, pluggy-0.13.1`

Tracebacks:
```
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 337, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 52, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 335, in <lambda>
INTERNALERROR>     outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 637, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 337, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 52, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 335, in <lambda>
INTERNALERROR>     outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\plugin.py", line 119, in modify_items
INTERNALERROR>     sorter = Sorter(config, items)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\sorter.py", line 56, in __init__
INTERNALERROR>     self.settings: Settings = Settings(config)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\settings.py", line 63, in __init__
INTERNALERROR>     and auto_mark_dep.lower() in ("1", "yes", "y", "true", "t", "on")
INTERNALERROR> AttributeError: 'bool' object has no attribute 'lower'
```
```
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 337, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 52, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 335, in <lambda>
INTERNALERROR>     outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "C:\Python38\lib\site-packages\_pytest\main.py", line 637, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 337, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 52, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 335, in <lambda>
INTERNALERROR>     outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Python38\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\plugin.py", line 120, in modify_items
INTERNALERROR>     items[:] = sorter.sort_items()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\sorter.py", line 77, in sort_items
INTERNALERROR>     self.collect_markers()
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\sorter.py", line 295, in collect_markers
INTERNALERROR>     self.mark_binning(item, dep_marks, aliases)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\sorter.py", line 131, in mark_binning
INTERNALERROR>     self.handle_order_mark(item)
INTERNALERROR>   File "C:\Python38\lib\site-packages\pytest_order\sorter.py", line 165, in handle_order_mark
INTERNALERROR>     order = mark.args[0] if mark.args else mark.kwargs.get("index")
INTERNALERROR> AttributeError: 'NoneType' object has no attribute 'args'
```

You can get more by just running tests from the `pytest-dependency` project while having `pytest-order` installed.


Discussion / hints:
Thanks, interesting - will check this! Looks like `iter_markers` can return `None` in the list, I wouldn't have expected this. This is with your PR branch, right?
Hm, I couldn't reproduce this. In the develop branch, there are no errors in the tests, and in your branch, there are several tests failing with `INTERNALERROR> AttributeError: 'bool' object has no attribute 'lower'`, seems like `auto_mark_dep` has changed from `str` to `bool` or something.

Ah, I had tested with my `main` branch, not with `1.0.0` - I can reproduce this with the released version (one test failing with that error). So, this is fixed, but not released yet - probably will make a patch release in this case.
I just noticed that you also have the other error (`'bool' object has no attribute 'lower'`) in your error listing, I had missed that. ~That only happens in your branch, though.~ 
Turns out I didn't have the develop version of `pytest-dependency` installed, but the released version. This does not happen in the released version - obviously the type of the option has changed in `develop`.
Yeah, well, this was tested with latest releases of both `pytest-order` and `pytest-dependency` available from PyPi for Python 3.8, so that'd be `pytest-order==1.0.0` and `pytest-dependency==0.5.1`. But I see that you've figured out most of it already.
Yes - the first problem only happens in the development branch of `pytest-dependency`, due to the fix of [this issue](https://github.com/RKrahl/pytest-dependency/pull/58) - I fixed this to work with both versions, will make a PR soon. The second issue was already fixed in `main`, as mentioned.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e210d33..398c6b3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,9 @@
 ### Fixes
 - correctly handle combined class and test order markers,
   see [#45](https://github.com/pytest-dev/pytest-order/issues/45)
+- adapt code to changed type of `automark_dependency` in development branch
+  of `pytest-dependency`,
+  see [#58](https://github.com/pytest-dev/pytest-order/issues/58)
 
 ## [Version 1.0.0](https://pypi.org/project/pytest-order/1.0.0/) (2021-05-30)
 First Python 3 only version.
diff --git a/README.md b/README.md
index 4c0b74e..a22bc94 100644
--- a/README.md
+++ b/README.md
@@ -21,13 +21,34 @@ Documentation
 Apart from this overview, the following information is available:
 - usage documentation for the [latest release](https://pytest-dev.github.io/pytest-order/stable/)
 - usage documentation for the [current main branch](https://pytest-dev.github.io/pytest-order/dev/)
-- all examples shown in the documentation can also be found in the 
+- most examples shown in the documentation can also be found in the 
   [repository](https://github.com/pytest-dev/pytest-order/tree/main/example)
 - the [Release Notes](https://github.com/pytest-dev/pytest-order/blob/main/CHANGELOG.md)
   with a list of changes in the latest versions
 - a [list of open issues](https://github.com/pytest-dev/pytest-order/blob/main/old_issues.md)
   in the original project and their handling in `pytest-order`
 
+Features
+--------
+`pytest-order` provides the following features:
+- ordering of tests [by index](https://pytest-dev.github.io/pytest-order/stable/usage.html#ordering-by-numbers)
+- ordering of tests both from the start and from the end (via negative
+  index)
+- ordering of tests [relative to each other](https://pytest-dev.github.io/pytest-order/stable/usage.html#order-relative-to-other-tests)
+  (via the `before` and `after` marker attributes)
+- session-, module- and class-scope ordering via the
+  [order-scope](https://pytest-dev.github.io/pytest-order/stable/configuration.html#order-scope) option
+- directory scope ordering via the
+  [order-scope-level](https://pytest-dev.github.io/pytest-order/stable/configuration.html#order-scope-level) option
+- hierarchical module and class-level ordering via the 
+  [order-group-scope](https://pytest-dev.github.io/pytest-order/stable/configuration.html#order-group-scope) option
+- ordering tests with `pytest-dependency` markers if using the
+  [order-dependencies](https://pytest-dev.github.io/pytest-order/stable/configuration.html#order-dependencies) option, 
+  more information about `pytest-dependency` compatibility
+  [here](https://pytest-dev.github.io/pytest-order/stable/other_plugins.html#relationship-with-pytest-dependency) 
+- sparse ordering of tests via the 
+  [sparse-ordering](https://pytest-dev.github.io/pytest-order/stable/configuration.html#sparse-ordering) option
+
 Overview
 --------
 _(adapted from the original project)_
@@ -70,25 +91,6 @@ yields the output:
 
     =========================== 2 passed in 0.01 seconds ===========================
 
-Features
---------
-`pytest-order` provides the following features:
-- ordering of tests by index, as shown above
-- ordering of tests both from the start and from the end (via negative
-  index)
-- ordering of tests relative to each other (via the `before` and `after`
-  marker attributes) 
-- session-, module- and class-scope ordering via the ``order-scope`` option
-- directory scope ordering via the ``order-scope-level`` option
-- hierarchical module and class-level ordering via the ``group-order-scope``
-  option
-- ordering tests with `pytest-dependency` markers if using the
-  ``order-dependencies`` option
-- sparse ordering of tests via the ``sparse-ordering`` option
-  
-A usage guide for each feature can be
-found in the [documentation](https://pytest-dev.github.io/pytest-order/dev/).
-
 Contributing
 ------------
 Contributions are very welcome. Tests can be run with 
diff --git a/tox.ini b/tox.ini
index 365c296..7dcb61f 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,7 +1,7 @@
 [tox]
 envlist =
     {py36,py37,py38,py39,pypy3}-pytest{50,51,52,53,54,60,61,62}
-    {py310dev}-pytest{624}
+    {py310}-pytest{624}
 [testenv]
 deps =
     pytest50: pytest>=5.0,<5.1


Test patch:
diff --git a/.github/workflows/pythontests.yml b/.github/workflows/pythontests.yml
index 5ae9398..7f2de4a 100644
--- a/.github/workflows/pythontests.yml
+++ b/.github/workflows/pythontests.yml
@@ -40,7 +40,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest]
-        python-version: [3.6, 3.7, 3.8, 3.9, 3.10-dev, pypy3]
+        python-version: [3.6, 3.7, 3.8, 3.9, "3.10", pypy3]
         exclude:
           - os: windows-latest
             python-version: pypy3
diff --git a/pytest_order/settings.py b/pytest_order/settings.py
index c6ef919..9085769 100644
--- a/pytest_order/settings.py
+++ b/pytest_order/settings.py
@@ -56,9 +56,10 @@ def __init__(self, config: Config) -> None:
             self.group_scope = self.scope
         try:
             auto_mark_dep = config.getini("automark_dependency")
+            if isinstance(auto_mark_dep, str):
+                auto_mark_dep = auto_mark_dep.lower() in (
+                    "1", "yes", "y", "true", "t", "on"
+                )
         except ValueError:
             auto_mark_dep = False
-        self.auto_mark_dep = (
-            auto_mark_dep
-            and auto_mark_dep.lower() in ("1", "yes", "y", "true", "t", "on")
-        )
+        self.auto_mark_dep = auto_mark_dep