Repository: python/mypy

Issue:
str-bytes-safe with new Formatted string literals
with this code:

`f"{b'def'}"`

I get this from mypy:

`On Python 3 '{}'.format(b'abc') produces "b'abc'", not 'abc'; use '{!r}'.format(b'abc') if this is desired behavior  [str-bytes-safe]`

That message doesn't make it clear whether there's any issue with my code, because it's about a different string formatting.
And if it is an issue, it doesn't tell me what formatted string literal to use if this is desired behavior.


Discussion:
Should be fixed in `0.920` (or in `0.930`). Works correctly on `master`:
<img width="752" alt="Снимок экрана 2021-12-22 в 12 27 10" src="https://user-images.githubusercontent.com/4660275/147069644-6f529e65-39fd-486b-9012-a7e02c1a4e13.png">

That still doesn't make sense for an output message.
How is someone supposed to use `"{!r}"` with bytes in a formatted string literal?
We're not formatting `b'abc'` with `{}`, because the bytes are **inside** the curly braces.
Any suggestions?
What is the code that you would suggest someone replace this with, if they want the output that this gives?

```python3
b = b'abc'
s = f"{b}"
print(s)
```

I think my suggestion would be that `[str-bytes-safe]` only applies to this:

`"{}".format(b)`

and should not say anything about this:

`f"{b}"`
Why is this closed? I still don't know how to avoid the warning if I want to keep using f-strings.

Also why does the warning exist in the first place, I'd expect bytes to be printed as the b"..." honestly and it seems weird for a type checking tool to warn me about how formatting works.
This error message is completely unclear and unactionable.
@ajkaijanaho do you have suggestions on how to improve it? :)
@Syndace It has a distinct error code `str-bytes-safe` so you can disable it.

This should illustrate how to take the action on this error:
https://mypy-play.net/?mypy=latest&python=3.11&gist=097c14bfe36a2f09487802689904d5f0

@sobolevn I think 'abc' might be confusing, would be nicer if we could "unparse" the user's actual expression there. For example, in my playground that'd be:
```diff
-main.py:3: error: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior  [str-bytes-safe]
-main.py:5: error: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior  [str-bytes-safe]
+main.py:3: error: On Python 3 formatting "data" with "{}" produces "b'...'", not "..."; use "{!r}" if this is desired behavior  [str-bytes-safe]
+main.py:5: error: On Python 3 formatting "{data}" produces "b'...'", not "..."; use "{data!r}" if this is desired behavior  [str-bytes-safe]
```

Another thought would be to remove "On Python 3" since I think we're sufficiently far from Python 2.
> @ajkaijanaho do you have suggestions on how to improve it? :)

I do not understand it, so no. It is not clear to me why it is something that needs to be flagged by mypy, and what problem it solves. Certainly, in the context of where I ran into it, the message was incomprehensible, and Googling came up with no useful documentation (but it did turn up this issue).

> I think 'abc' might be confusing, would be nicer if we could "unparse" the user's actual expression there. For example, in my playground that'd be:

Yes, this makes it clearer, but it does not answer what the recommended action is if that is not the desired behavior.
This rule was made as a result of transitioning from Python 2 to Python 3.

f-strings (https://peps.python.org/pep-0498/) never existed in Python 2.

There's no reason for this rule to say anything about f-strings.
> Another thought would be to remove "On Python 3" since I think we're sufficiently far from Python 2.

I think removing "on Python 3" would make it MORE confusing, because that would remove context of why the rule exists.
This rule is only relevant in a transition from Python 2 to Python 3.

There are still some people migrating code bases from Python 2 to Python 3. For those people, this rule could help catch an error.

For anyone else, this rule is irrelevant - as pointed out above:

https://github.com/python/mypy/issues/11806#issuecomment-1081612521

> Also why does the warning exist in the first place, I'd expect bytes to be printed as the b"..." honestly and it seems weird for a type checking tool to warn me about how formatting works.

https://github.com/python/mypy/issues/11806#issuecomment-1479562592

> It is not clear to me why it is something that needs to be flagged by mypy, and what problem it solves.

This rule is just not relevant outside of a transition from Python 2 to Python 3.
If you think we're sufficiently far from Python 2, that means it's time to remove this rule from mypy.
In a pure Python 3 context, a helpful error message might be something that outlines your options e.g.

`"{data}"` (or equivalent `"{}".format(data)`) is implicitly getting a `repr` of the bytes.
- If that's your intention, make it explicit by adding `!r` to the format string.
- If you meant to decode the `bytes` to `str`, call `data.decode()`.

Now, whether "printing bytes as if they were a string" is a likely error (especially for devs who didn't come from py2) or mypy nannying the developer is one that's hard to answer. I do think adding `!r` better complies with "explicit is better than implicit" :)
> I do think adding `!r` better complies with "explicit is better than implicit"

Wouldn't that mean that adding `!r` is better for every data type whose `__str__` gives the same result as `__repr__`?

`f"1 2 {3!r}"`

How do we figure out which data types mypy should suggest `!r` with, and which data types it shouldn't suggest `!r` with?
I think it's meant to be a heuristic, i.e. you'd do it for types where you're likely to spot a user error. (Generally, mypy often makes pragmatic decisions, seeing it more as a tool for error-checking than an abstract type-checker.)
I think this is only likely to spot a user error in the context of transitioning from Python 2 to Python 3.
Outside of that context, it's just more likely to cause confusion, because `bytes` is just like a lot of other data types, where `__str__` is the same as `__repr__`.
Since we no longer support Python2, do we even need this feature? 🤔 

There are still some people migrating code bases from Python2 to Python3, so it could have a little usefulness for those people.
But I wouldn't think it's worth it to keep maintaining this feature.
People dealing with those migrations are going to have to deal with similar issues on their own anyway.
> There are still some people migrating code bases from Python2 to Python3, so it could have a little usefulness for those people.

I think for them using older mypy versions would be better, because older versions support both 2 and 3. It is easier to use during the transition.

I will send a PR removing this feature. Any feedback is welcome!
To be clear, this feature is not for analyzing Python 2 code, so supporting Python 2 isn't really the issue.
It is definitely for Python 3 specifically.

But it's for analyzing Python 3 code:
 - that was migrated from Python 2
 - that was written by people who were used to Python 2, so they follow a pattern they used in Python 2
 - for people who find Python 2 code online, and copy/paste it into their Python 3 project
 - etc.

Some of these will never stop happening. So there will always be some usefulness,, but is it worth it?
> In a pure Python 3 context, a helpful error message might be something that outlines your options e.g.
> 
> `"{data}"` (or equivalent `"{}".format(data)`) is implicitly getting a `repr` of the bytes.
> 
> * If that's your intention, make it explicit by adding `!r` to the format string.
> * If you meant to decode the `bytes` to `str`, call `data.decode()`.
> 
> Now, whether "printing bytes as if they were a string" is a likely error (especially for devs who didn't come from py2) or mypy nannying the developer is one that's hard to answer. I do think adding `!r` better complies with "explicit is better than implicit" :)

I think. in the rare case where this actually comes up, this advice would be sound. Sometimes people are not really realizing they have a bytes object rather than str. But as this is not a type error (as in, something that would raise an exception at runtime at least some of the time), it's not really something that mypy needs to deal with, and Python is full of such sometimes surprising behavior, bytes being nothing special in that regard.  I would not object to removing this check, but if it is kept, I think the issue should be discussed in the docs (with a pointer in the error message).
>  I think the issue should be discussed in the docs (with a pointer in the error message).

I agree, and I think that this is the best way to move forward. We should document the error code [here](https://mypy.readthedocs.io/en/stable/error_code_list.html) where we discuss other error codes, and provide example error messages and discuss why we have this check. If this still causing confusion in the future, we can consider turning off the error code by default, but I think that this should happen only some time after we've made the documentation updates.

I also think that forgetting to explicitly encode bytes values is still a somewhat common issue in code that was never intended to be run on Python 2, and mypy catching these can be helpful. The open issue is whether this check should be enabled by default.
Ok, fair enough! I think that judging from beginner's perspective this check can actually help:
- All types can be converted to `str` as simple as `str(your_object)`
- Except for bytes, where you need to use `.decode(encoding)`

I will change my PR to include better docs. Thanks a lot for your feedback, everyone!
> * Except for bytes

This is not an exception. `bytes` can also be converted to `str` as simple as `str(your_bytes)`

This conversion is often what is desired and there's no reason to add `!r` if this is what is desired.
"This conversion is often what is desired" is precisely what's left to be debated: the basis of any lint rule is the ratio of detected errors vs. nuisance to users.

The 'nuisance to users' is what you need to do to mitigate (in this case, adding two characters, which is small relative to other nuisances like adding `# type: ignore`, `assert ...` etc.).

If 'detected errors' is close to nil, then any amount of nuisance is too much, of course.

Resolving patch:
diff --git a/mypy/checkstrformat.py b/mypy/checkstrformat.py
--- a/mypy/checkstrformat.py
+++ b/mypy/checkstrformat.py
@@ -435,9 +435,9 @@ def perform_special_format_checks(
                 actual_type, "__str__"
             ):
                 self.msg.fail(
-                    'On Python 3 formatting "b\'abc\'" with "{}" '
-                    'produces "b\'abc\'", not "abc"; '
-                    'use "{!r}" if this is desired behavior',
+                    'If x = b\'abc\' then f"{x}" or "{}".format(x) produces "b\'abc\'", '
+                    'not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). '
+                    "Otherwise, decode the bytes",
                     call,
                     code=codes.STR_BYTES_PY3,
                 )
@@ -946,9 +946,8 @@ def check_s_special_cases(self, expr: FormatStringExpr, typ: Type, context: Cont
             # Couple special cases for string formatting.
             if has_type_component(typ, "builtins.bytes"):
                 self.msg.fail(
-                    'On Python 3 formatting "b\'abc\'" with "%s" '
-                    'produces "b\'abc\'", not "abc"; '
-                    'use "%r" if this is desired behavior',
+                    'If x = b\'abc\' then "%s" % x produces "b\'abc\'", not "abc". '
+                    'If this is desired behavior use "%r" % x. Otherwise, decode the bytes',
                     context,
                     code=codes.STR_BYTES_PY3,
                 )


Test patch:
diff --git a/mypyc/test-data/run-strings.test b/mypyc/test-data/run-strings.test
--- a/mypyc/test-data/run-strings.test
+++ b/mypyc/test-data/run-strings.test
@@ -232,8 +232,7 @@ def test_fstring_basics() -> None:
 
     x = bytes([1, 2, 3, 4])
     # assert f'bytes: {x}' == "bytes: b'\\x01\\x02\\x03\\x04'"
-    # error: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc";
-    #        use "{!r}" if this is desired behavior behavior
+    # error: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
 
     float_num = 123.4
     assert f'{float_num}' == '123.4'
diff --git a/test-data/unit/check-errorcodes.test b/test-data/unit/check-errorcodes.test
--- a/test-data/unit/check-errorcodes.test
+++ b/test-data/unit/check-errorcodes.test
@@ -638,8 +638,8 @@ def g() -> int:
 '%d' % 'no'  # E: Incompatible types in string interpolation (expression has type "str", placeholder has type "Union[int, float, SupportsInt]")  [str-format]
 '%d + %d' % (1, 2, 3)  # E: Not all arguments converted during string formatting  [str-format]
 
-'{}'.format(b'abc')  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior  [str-bytes-safe]
-'%s' % b'abc'  # E: On Python 3 formatting "b'abc'" with "%s" produces "b'abc'", not "abc"; use "%r" if this is desired behavior  [str-bytes-safe]
+'{}'.format(b'abc')  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes  [str-bytes-safe]
+'%s' % b'abc'  # E: If x = b'abc' then "%s" % x produces "b'abc'", not "abc". If this is desired behavior use "%r" % x. Otherwise, decode the bytes  [str-bytes-safe]
 [builtins fixtures/primitives.pyi]
 [typing fixtures/typing-medium.pyi]
 
diff --git a/test-data/unit/check-formatting.test b/test-data/unit/check-formatting.test
--- a/test-data/unit/check-formatting.test
+++ b/test-data/unit/check-formatting.test
@@ -30,8 +30,8 @@ xb: bytes
 xs: str
 
 '%s' % xs   # OK
-'%s' % xb   # E: On Python 3 formatting "b'abc'" with "%s" produces "b'abc'", not "abc"; use "%r" if this is desired behavior
-'%(name)s' % {'name': b'value'}  # E: On Python 3 formatting "b'abc'" with "%s" produces "b'abc'", not "abc"; use "%r" if this is desired behavior
+'%s' % xb   # E: If x = b'abc' then "%s" % x produces "b'abc'", not "abc". If this is desired behavior use "%r" % x. Otherwise, decode the bytes
+'%(name)s' % {'name': b'value'}  # E: If x = b'abc' then "%s" % x produces "b'abc'", not "abc". If this is desired behavior use "%r" % x. Otherwise, decode the bytes
 [builtins fixtures/primitives.pyi]
 
 [case testStringInterpolationCount]
@@ -435,21 +435,21 @@ N = NewType('N', bytes)
 n: N
 
 '{}'.format(a)
-'{}'.format(b)  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
-'{}'.format(x)  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
-'{}'.format(n)  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
+'{}'.format(b)  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
+'{}'.format(x)  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
+'{}'.format(n)  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
 
-f'{b}'  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
-f'{x}'  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
-f'{n}'  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
+f'{b}'  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
+f'{x}'  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
+f'{n}'  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
 
 class C(Generic[B]):
     x: B
     def meth(self) -> None:
-        '{}'.format(self.x)  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
+        '{}'.format(self.x)  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
 
 def func(x: A) -> A:
-    '{}'.format(x)  # E: On Python 3 formatting "b'abc'" with "{}" produces "b'abc'", not "abc"; use "{!r}" if this is desired behavior
+    '{}'.format(x)  # E: If x = b'abc' then f"{x}" or "{}".format(x) produces "b'abc'", not "abc". If this is desired behavior, use f"{x!r}" or "{!r}".format(x). Otherwise, decode the bytes
     return x
 
 '{!r}'.format(a)