Repository: psf/black

Issue:
s390x: test_python2/test_python2_unicode_literals can't assign to () INTERNAL ERROR
During the build of 19.10b0 in Fedora, the following test failure occurs on s390x (Big Endian) architecture:

```
======================================================================
FAIL: test_python2 (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/black-19.10b0/black.py", line 3754, in assert_equivalent
    src_ast = parse_ast(src)
  File "/builddir/build/BUILD/black-19.10b0/black.py", line 3686, in parse_ast
    return ast27.parse(src)
  File "/usr/lib64/python3.8/site-packages/typed_ast/ast27.py", line 50, in parse
    return _ast27.parse(source, filename, mode)
  File "<unknown>", line 10
SyntaxError: can't assign to ()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/unittest/mock.py", line 1342, in patched
    return func(*newargs, **newkeywargs)
  File "/builddir/build/BUILD/black-19.10b0/tests/test_black.py", line 543, in test_python2
    black.assert_equivalent(source, actual)
  File "/builddir/build/BUILD/black-19.10b0/black.py", line 3756, in assert_equivalent
    raise AssertionError(
AssertionError: cannot use --safe with this file; failed to parse source file.  AST error message: can't assign to () (<unknown>, line 10)
======================================================================
FAIL: test_python2_unicode_literals (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.8/unittest/mock.py", line 1342, in patched
    return func(*newargs, **newkeywargs)
  File "/builddir/build/BUILD/black-19.10b0/tests/test_black.py", line 560, in test_python2_unicode_literals
    black.assert_equivalent(source, actual)
  File "/builddir/build/BUILD/black-19.10b0/black.py", line 3775, in assert_equivalent
    raise AssertionError(
AssertionError: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: 
--- src
+++ dst
@@ -1,4 +1,70 @@
 Module(
   body=
+    ImportFrom(
+      level=
+        0,  # int
+      module=
+        '__future__',  # str
+      names=
+        alias(
+          asname=
+            '_unicode_literals',  # str
+          name=
+            'unicode_literals',  # str
+        )  # /alias
+    )  # /ImportFrom
+    ImportFrom(
+      level=
+        0,  # int
+      module=
+        '__future__',  # str
+      names=
+        alias(
+          asname=
+            None,  # NoneType
+          name=
+            'absolute_import',  # str
+        )  # /alias
+    )  # /ImportFrom
+    ImportFrom(
+      level=
+        0,  # int
+      module=
+        '__future__',  # str
+      names=
+        alias(
+          asname=
+            'lol',  # str
+          name=
+            'print_function',  # str
+        )  # /alias
+        alias(
+          asname=
+            None,  # NoneType
+          name=
+            'with_function',  # str
+        )  # /alias
+    )  # /ImportFrom
+    Expr(
+      value=
+        Constant(
+          value=
+            'hello',  # str
+        )  # /Constant
+    )  # /Expr
+    Expr(
+      value=
+        Constant(
+          value=
+            'hello',  # str
+        )  # /Constant
+    )  # /Expr
+    Expr(
+      value=
+        Constant(
+          value=
+            'hello',  # str
+        )  # /Constant
+    )  # /Expr
   type_ignores=
 )  # /Module
----------------------------------------------------------------------
Ran 119 tests in 18.012s
FAILED (failures=2)
```

**To Reproduce**, run the test suite on s390x. 

Here is the build log with all the commands: [build.log](https://github.com/psf/black/files/3782557/build.log)

Here is the root log with all the package versions: [root.log](https://github.com/psf/black/files/3782561/root.log)


**Expected behavior**
Test succeed an all architectures.

**Environment:**

- Version: 19.10b0
- OS and Python version: Linux, Fedora 32 on s390x, Python 3.8.0

**Does this bug also happen on master?** yes, on 6bedb5c58a7d8c25aa9509f8217bc24e9797e90d

**Additional context** The problem does not happen on the same build system with armv7hl or ppc64le.



Discussion:
On master 6bedb5c58a7d8c25aa9509f8217bc24e9797e90d:

```
======================================================================
FAIL: test_python2 (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/black-master/black.py", line 3754, in assert_equivalent
    src_ast = parse_ast(src)
  File "/builddir/build/BUILD/black-master/black.py", line 3686, in parse_ast
    return ast27.parse(src)
  File "/usr/lib64/python3.8/site-packages/typed_ast/ast27.py", line 50, in parse
    return _ast27.parse(source, filename, mode)
  File "<unknown>", line 10
SyntaxError: can't assign to ()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/unittest/mock.py", line 1342, in patched
    return func(*newargs, **newkeywargs)
  File "/builddir/build/BUILD/black-master/tests/test_black.py", line 543, in test_python2
    black.assert_equivalent(source, actual)
  File "/builddir/build/BUILD/black-master/black.py", line 3756, in assert_equivalent
    raise AssertionError(
AssertionError: cannot use --safe with this file; failed to parse source file.  AST error message: can't assign to () (<unknown>, line 10)
======================================================================
FAIL: test_python2_unicode_literals (tests.test_black.BlackTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.8/unittest/mock.py", line 1342, in patched
    return func(*newargs, **newkeywargs)
  File "/builddir/build/BUILD/black-master/tests/test_black.py", line 560, in test_python2_unicode_literals
    black.assert_equivalent(source, actual)
  File "/builddir/build/BUILD/black-master/black.py", line 3775, in assert_equivalent
    raise AssertionError(
AssertionError: INTERNAL ERROR: Black produced code that is not equivalent to the source.  Please report a bug on https://github.com/psf/black/issues.  This diff might be helpful: 
--- src
+++ dst
@@ -1,4 +1,70 @@
 Module(
   body=
+    ImportFrom(
+      level=
+        0,  # int
+      module=
+        '__future__',  # str
+      names=
+        alias(
+          asname=
+            '_unicode_literals',  # str
+          name=
+            'unicode_literals',  # str
+        )  # /alias
+    )  # /ImportFrom
+    ImportFrom(
+      level=
+        0,  # int
+      module=
+        '__future__',  # str
+      names=
+        alias(
+          asname=
+            None,  # NoneType
+          name=
+            'absolute_import',  # str
+        )  # /alias
+    )  # /ImportFrom
+    ImportFrom(
+      level=
+        0,  # int
+      module=
+        '__future__',  # str
+      names=
+        alias(
+          asname=
+            'lol',  # str
+          name=
+            'print_function',  # str
+        )  # /alias
+        alias(
+          asname=
+            None,  # NoneType
+          name=
+            'with_function',  # str
+        )  # /alias
+    )  # /ImportFrom
+    Expr(
+      value=
+        Constant(
+          value=
+            'hello',  # str
+        )  # /Constant
+    )  # /Expr
+    Expr(
+      value=
+        Constant(
+          value=
+            'hello',  # str
+        )  # /Constant
+    )  # /Expr
+    Expr(
+      value=
+        Constant(
+          value=
+            'hello',  # str
+        )  # /Constant
+    )  # /Expr
   type_ignores=
 )  # /Module
----------------------------------------------------------------------
Ran 119 tests in 18.123s
FAILED (failures=2)
```
First I was like LOLWAT but then I was like LOLWAT.

Big endian failing on Black which is itself pure Python and doing relatively little in terms of numeric computation anyway suggests to me that this is probably a typed-ast bug. I have no proof as I don't have an environment to test this in. Miro, do you have an idea where I can get access to a s390x box to look at this?

(also, to be fair, I should mention this will probably land under low-pri)
I don't have an easy to access environment with this, sorry. Low prio for me as well, don't worry.
FWIW, Travis supports `s390x` now, using that in some open source projects.

But given `black` is pure Python, maybe it's a CPython bug indeed :slightly_smiling_face: 
You can also use the openSUSE build service. Currently the mentioned tests fail for s390x and ppc64:
https://build.opensuse.org/package/show/devel:languages:python/python-black

(I am about to send a submit request that skips the tests in question on big endian platforms. But with an account, you can always branch the package and modify the specfile)
Here's a branch to test s390x on Travis CI:

* https://github.com/hugovk/black/commit/d01ade49194fede80ba5803b7083a0f184504825

But it's erroring installing dependencies:

* "Could not build wheels for yarl, multidict which use PEP 517 and cannot be installed directly"
* https://travis-ci.org/github/hugovk/black/jobs/720892873

Probably related: https://github.com/python/typed_ast/issues/139
FWIW, I had no issues installing dependencies on a QEMU-emulated `s390x`.

Running roughly along the lines of
```
docker run -it --rm --privileged multiarch/qemu-user-static --reset --persistent yes

docker run -it --rm s390x/ubuntu:20.04

apt-get update
apt-get install -y build-essential git python3 python3-dev python-is-python3
apt-get install -y python3-pip
# One can alternatively bootstrap from PyPa in case of any issues
pip3 install -U pip
pip install pipenv pytest

# git clone black and change to its dir etc...

pipenv install --dev
# pipenv shell doesn't quite work, activating manually
source ~/.local/share/virtualenvs/black-Jzvey4U1/bin/activate
python -m unittest
```

Two tests fail:
```
<...>
----------------------------------------------------------------------
Ran 148 tests in 203.234s

FAILED (failures=1, errors=1)
```

(Full output: https://gist.github.com/vytas7/e4d98d66f79e9b4b2307815ca5782985 )
Hmm, yours are different tests than those failing on openSUSE and Fedora
This should be fixed by https://github.com/python/typed_ast/pull/152 (confirmed in https://github.com/python/typed_ast/issues/151#issuecomment-752342743)

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -71,7 +71,7 @@ def get_long_description() -> str:
         "click>=7.1.2",
         "appdirs",
         "toml>=0.10.1",
-        "typed-ast>=1.4.0",
+        "typed-ast>=1.4.2",
         "regex>=2020.1.8",
         "pathspec>=0.6, <1",
         "dataclasses>=0.6; python_version < '3.7'",


Test patch: