Repository: pytest-dev/pytest
Pull request: Fix `disable_test_id_escaping_and_forfeit_all_rights_to_community_support` option when using `pytest.param(..., id="...")`
Closes #9037 .
The additional test cases were implemented in almost the same way as the existing similar test cases.

- [X] Include documentation when adding new features.
- [X] Include new tests or update existing tests when applicable.
- [X] Allow maintainers to push and squash when merging my commits. Please uncheck this if you prefer to squash the commits yourself.
- [X] Add text like ``closes #XYZW`` to the PR description and/or commits (where ``XYZW`` is the issue number). See the [github docs](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) for more information.
- [X] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](https://github.com/pytest-dev/pytest/blob/main/changelog/README.rst) for details.
- [X] Add yourself to `AUTHORS` in alphabetical order.


Problem statement:
Parametrized tests don't display unicode in ids correctly.
```python
@pytest.mark.parametrize(
    "string1", [
        pytest.param("test1", id="unicode in id い")
])
def test_unicode_in_param_id(string1:str) -> None:
    assert string1 == "test1"

```

running it with `pytest -v test.py` gives me

```
platform darwin -- Python 3.9.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/solarmist/repos/pytest
collected 1 item

test.py::test_unicode_in_param_id[unicode in id \u3044] PASSED                                                                                                                                                                                         [100%]
```

I expected the [] to show `unicode in id い`

`pip list`

```Package    Version
---------- -------
pip        21.2.4
setuptools 57.4.0
wheel      0.36.2```

This is on a macOS 11.4.

Hints from issue discussion:
I just saw this message in the docs, but in 2021 this does not feel reasonable to me anymore. Unicode should have 1st class support everywhere nowadays. Almost anyone using python for business applications needs some level of international text support.

I can understand plugins and OS display issues still being an issue and for the warning to exist for the forseeable future, but  pytest itself should fully support unicode without "forfeit[ing] all rights to community support".

> Note
> pytest by default escapes any non-ascii characters used in unicode strings for the parametrization because it has several downsides. If however you would like to use unicode strings in parametrization and see them in the terminal as is (non-escaped), use this option in your pytest.ini:
> 
> [pytest]
> disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True
> Keep in mind however that this might cause unwanted side effects and even bugs depending on the OS used and plugins currently installed, so use it at your own risk.
> 
That said I'm glad there's a workaround for the moment, but I still find it hard to believe such a key piece of the python ecosystem doesn't fully support unicode with how international python is.

The issue that gave rise to this
https://github.com/pytest-dev/pytest/issues/5294
i can understand the sentiment, but regular severe issues with different platforms, operating systems and encoding references as well as regular issues with ascii lookalike unicode characters speak a language of necessary confusion prevention


while modern pythons seem to do better, nobody has stepped up to actually correctly test and as required in this case exhaustively validate correctness

for example on osx you will get Unicode normalization based file-name conflicts when using non-normalized Unicode,
there is similar structural horrors on different types of windows apis, and then there is posix filenames, which are just messy blobs

and then there is surrogate escapes in python, which don't transfer sanely to some apis

while i'd be happy for 2021 style test ids, technology constraints from the 1980's are pretty much still ruining all the fun, and i dont see anyone stepping up to do the hard painful detail work of ensuring it works everywhere 

and without that unicode in the ids is just a time bomb to mess things up for the users
I can understand and empathize with that, but at the same the scorched earth "Lasciate ogne speranza, voi ch'intrate " (abandon hope all ye who enter here) message feels inappropriate.

How can we start building towards that mutually desirable end goal?  I'm thinking of something like a master issue that captures failure cases and expected failure cases (the hard painful detail work, you mention) so at least there's hope of getting there (even if only for a subset of users in the near term) eventually.

I'm willing to start digging into it and chipping away at the problems bit by bit, but I'm woefully ignorant of the nitty-gritty at this point so I'll need all the help I can get. 

Would you or someone in the community be willing to help me write such a master ticket?


Also I put the option in my `pyproject.toml` file as

```
[tool.pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = true
```

and 

```
[tool.pytest.ini_options]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = true
```

And a `pytest.ini` file as
```
[pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True
```
but was unable to get any of those variations to display unescaped unicode.
if the option doesn't work, that's a bug

i'm happy to see any progress in this direction, i'm happy to even add a project about this

its not entirely clear what ha to happen to use node/test ids safely

we do want to investigate moving test ids from the string they are currently to something more structured and immutable that supports making text/fs save encodings out of a id/name
Completely agree with you. And I feel that making the project to capture start collecting requirements, concerns and issues is the right first step.

Sure, I'd suggest that the project be considered to be tracking the latest dev until its at a sufficiently advanced state that people aren't worried about it anymore.

My thoughts are that the project will end up being a long list of bugs, edge cases, or known failure modes which would be unreasonable to try and work around that will be built bit by bit.
in github terms thats possibly more like a milestone + a accompanying project to do kanban/triage


Great! Thank you for hearing me out.
i'd like to note that i'm currently very short on time, so it may be a little while before i can get to bootstrap this effort,
we can certainly create a milestone/project by the end of the week and declare some goals for it so the details can progress

Commit message: Update changelog/9037.bugfix.rst

Co-authored-by: Ronny Pfannschmidt <opensource@ronnypfannschmidt.de>

Patch to review:
diff --git a/AUTHORS b/AUTHORS
index 1ee868448d4..172ce3cf715 100644
--- a/AUTHORS
+++ b/AUTHORS
@@ -406,6 +406,7 @@ Sviatoslav Sydorenko
 Sylvain Marié
 Tadek Teleżyński
 Takafumi Arakaki
+Takumi Otani
 Taneli Hukkinen
 Tanvi Mehta
 Tanya Agarwal
diff --git a/changelog/9037.bugfix.rst b/changelog/9037.bugfix.rst
new file mode 100644
index 00000000000..e4e7fabdd5f
--- /dev/null
+++ b/changelog/9037.bugfix.rst
@@ -0,0 +1 @@
+apply test id escaping opt out as configured by the ``disable_test_id_escaping_and_forfeit_all_rights_to_community_support`` option to ids from  ``pytest.param(..., id="...")`` in parametrized tests.
diff --git a/src/_pytest/mark/structures.py b/src/_pytest/mark/structures.py
index 92ade55f7c0..fade10728cf 100644
--- a/src/_pytest/mark/structures.py
+++ b/src/_pytest/mark/structures.py
@@ -21,7 +21,6 @@
 import warnings
 
 from .._code import getfslineno
-from ..compat import ascii_escaped
 from ..compat import NOTSET
 from ..compat import NotSetType
 from _pytest.config import Config
@@ -92,7 +91,6 @@ def param(
         if id is not None:
             if not isinstance(id, str):
                 raise TypeError(f"Expected id to be a string, got {type(id)}: {id!r}")
-            id = ascii_escaped(id)
         return cls(values, marks, id)
 
     @classmethod
diff --git a/src/_pytest/python.py b/src/_pytest/python.py
index 9c54dd20f80..dde442e8eda 100644
--- a/src/_pytest/python.py
+++ b/src/_pytest/python.py
@@ -924,7 +924,7 @@ def _resolve_ids(self) -> Iterable[str]:
         for idx, parameterset in enumerate(self.parametersets):
             if parameterset.id is not None:
                 # ID provided directly - pytest.param(..., id="...")
-                yield parameterset.id
+                yield _ascii_escaped_by_config(parameterset.id, self.config)
             elif self.ids and idx < len(self.ids) and self.ids[idx] is not None:
                 # ID provided in the IDs list - parametrize(..., ids=[...]).
                 yield self._idval_from_value_required(self.ids[idx], idx)
diff --git a/testing/python/metafunc.py b/testing/python/metafunc.py
index be224d9e20b..0a4ebf2c9af 100644
--- a/testing/python/metafunc.py
+++ b/testing/python/metafunc.py
@@ -625,6 +625,37 @@ def getini(self, name):
             ).make_unique_parameterset_ids()
             assert result == [expected]
 
+    def test_idmaker_with_param_id_and_config(self) -> None:
+        """Unit test for expected behavior to create ids with pytest.param(id=...) and
+        disable_test_id_escaping_and_forfeit_all_rights_to_community_support
+        option (#9037).
+        """
+
+        class MockConfig:
+            def __init__(self, config):
+                self.config = config
+
+            def getini(self, name):
+                return self.config[name]
+
+        option = "disable_test_id_escaping_and_forfeit_all_rights_to_community_support"
+
+        values: list[tuple[Any, str]] = [
+            (MockConfig({option: True}), "ação"),
+            (MockConfig({option: False}), "a\\xe7\\xe3o"),
+        ]
+        for config, expected in values:
+            result = IdMaker(
+                ("a",),
+                [pytest.param("string", id="ação")],
+                None,
+                None,
+                config,
+                None,
+                None,
+            ).make_unique_parameterset_ids()
+            assert result == [expected]
+
     def test_idmaker_duplicated_empty_str(self) -> None:
         """Regression test for empty strings parametrized more than once (#11563)."""
         result = IdMaker(


Code review comments:
diff_hunk: @@ -0,0 +1 @@
+apply test id escaping opt out as configured by the ``disable_test_id_escaping_and_forfeit_all_rights_to_community_support`` option to ids from  ``pytest.param(..., id="...")`` in parametrized tests.
path: changelog/9037.bugfix.rst
text: @user1:
```suggestion
Honor :confval:`disable_test_id_escaping_and_forfeit_all_rights_to_community_support` when escaping ids in parametrized tests.
```