Repository: kivy/python-for-android

Issue:
cffi UnicodeEncodeError: 'ascii' codec can't encode character '\u2018'
<!--
The issue tracker is a tool to address bugs.
Please use the Disord community or Stack Overflow for support questions,
more information at https://github.com/kivy/python-for-android#support

Before opening a new issue, make sure you do the following:
    * check that your issue isn't already filed: https://github.com/kivy/python-for-android/issues
    * prepare a short, runnable example that reproduces the issue
    * make sure to have `log_level = 2` in your `buildozer.spec`
    * reproduce the problem with the latest development version (`p4a.branch = master`)
    * double-check that the issue is indeed a bug and not a support request
    * please use backticks to format code or logs
-->

### Versions

* Python: host python3, target python3
* OS: Ubuntu 18.04
* Kivy: N/A
* Cython: 0.28.6

### Description

In a Docker environment, compiling `cffi` will fail. The compilation command used:
```sh
python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --bootstrap sdl2 --requirements python3,cffi
```

### buildozer.spec

Command:
```sh
python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --bootstrap sdl2 --requirements python3,cffi
```

### Logs

```
(venv) user@5bbd34e1299b:~/testapps$ p4a clean_recipe_build cffi
[INFO]:    Cleaning build for cffi recipe.
[INFO]:    Deleting /home/user/.local/share/python-for-android/build/other_builds/cffi-python3
(venv) user@5bbd34e1299b:~/testapps$ python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --bootstrap sdl2 --requirements python3,cffi
packages are []
running apk
WARNING: You did not supply an Android package identifier, trying org.test.testapp_python3_googlendk instead.
         This may fail if this is not a valid identifier
Including testapp/main.py
Including testapp/colours.png
[WARNING]: --ndk-version is deprecated and no longer necessary, the value you passed is ignored
[INFO]:    Will compile for the following archs: armeabi-v7a
[INFO]:    Getting Android API version from user argument: 27
[INFO]:    Available Android APIs are (19, 27)
[INFO]:    Requested API target 27 is available, continuing.
[INFO]:    Getting NDK dir from from user argument
[INFO]:    Found NDK revision 17.2.4988734
[INFO]:    Getting NDK API version (i.e. minimum supported API) from user argument
[INFO]:    Found virtualenv at /usr/bin/virtualenv
[INFO]:    Found the following toolchain versions: ['4.9']
[INFO]:    Picking the latest gcc toolchain, here 4.9
[INFO]:    No existing dists meet the given requirements!
[INFO]:    No dist exists that meets your requirements, so one will be built.
[INFO]:    Found a single valid recipe set: ['hostpython3', 'libffi', 'python3', 'setuptools', 'pycparser', 'cffi']
[INFO]:    The selected bootstrap is sdl2
[INFO]:    # Creating dist with sdl2 bootstrap
[INFO]:    Dist will have name bdisttest_python3_googlendk and recipes (python3, cffi)
[INFO]:    Dist will also contain modules () installed from pip
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Recipe build order is ['hostpython3', 'libffi', 'python3', 'setuptools', 'pycparser', 'cffi']
[INFO]:    # Downloading recipes 
[INFO]:    Downloading hostpython3
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/hostpython3
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/hostpython3
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    hostpython3 download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading libffi
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/libffi
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/libffi
[INFO]:    -> running basename https://github.com/libffi/libffi/archive/v3.2.1.tar.gz
[INFO]:    libffi download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading python3
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/python3
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/python3
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    python3 download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading setuptools
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/setuptools
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/setuptools
[INFO]:    -> running basename https://pypi.python.org/packages/source/s/setuptools/setuptools-40.0.0.zip
[INFO]:    setuptools download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading pycparser
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/pycparser
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/pycparser
[INFO]:    -> running basename https://pypi.python.org/packages/source/p/pycparser/pycparser-2.14.tar.gz
[INFO]:    pycparser download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Downloading cffi
[INFO]:    -> running mkdir -p /home/user/.local/share/python-for-android/packages/cffi
[INFO]:    -> directory context /home/user/.local/share/python-for-android/packages/cffi
[INFO]:    -> running basename https://pypi.python.org/packages/source/c/cffi/cffi-1.11.5.tar.gz
[INFO]:    cffi download already cached, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    # Building all recipes for arch armeabi-v7a
[INFO]:    # Unpacking recipes
[INFO]:    Unpacking hostpython3 for armeabi-v7a
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop
[INFO]:    hostpython3 is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking libffi for armeabi-v7a
[INFO]:    -> running basename https://github.com/libffi/libffi/archive/v3.2.1.tar.gz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/libffi/armeabi-v7a__ndk_target_21
[INFO]:    libffi is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking python3 for armeabi-v7a
[INFO]:    -> running basename https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/python3-libffi/armeabi-v7a__ndk_target_21
[INFO]:    python3 is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking setuptools for armeabi-v7a
[INFO]:    -> running basename https://pypi.python.org/packages/source/s/setuptools/setuptools-40.0.0.zip
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21
[INFO]:    setuptools is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking pycparser for armeabi-v7a
[INFO]:    -> running basename https://pypi.python.org/packages/source/p/pycparser/pycparser-2.14.tar.gz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/pycparser-python3/armeabi-v7a__ndk_target_21
[INFO]:    pycparser is already unpacked, skipping
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Unpacking cffi for armeabi-v7a
[INFO]:    -> running basename https://pypi.python.org/packages/source/c/cffi/cffi-1.11.5.tar.gz
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21
[INFO]:    -> running mv cffi-1.11.5 /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    <- directory context /home/user/testapps
[INFO]:    # Prebuilding recipes
[INFO]:    Prebuilding hostpython3 for armeabi-v7a
[INFO]:    hostpython3 has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding libffi for armeabi-v7a
[INFO]:    libffi has no prebuild_armeabi_v7a, skipping
[INFO]:    Applying patches for libffi[armeabi-v7a]
[INFO]:    libffi already patched, skipping
[INFO]:    Prebuilding python3 for armeabi-v7a
[INFO]:    python3 has no prebuild_armeabi_v7a, skipping
[INFO]:    Applying patches for python3[armeabi-v7a]
[INFO]:    python3 already patched, skipping
[INFO]:    Prebuilding setuptools for armeabi-v7a
[INFO]:    setuptools has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding pycparser for armeabi-v7a
[INFO]:    pycparser has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding cffi for armeabi-v7a
[INFO]:    cffi has no prebuild_armeabi_v7a, skipping
[INFO]:    Applying patches for cffi[armeabi-v7a]
[INFO]:    Applying patch disable-pkg-config.patch
[INFO]:    -> running patch -t -d /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi -p1 -i /home/user/pythonforandroid/recipes/cffi/disable-pkg-config.patch
[INFO]:    -> running touch /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi/.patched
[INFO]:    # Building recipes
[INFO]:    Building hostpython3 for armeabi-v7a
[INFO]:    Skipping hostpython3 (3.7.1) build, as it has already been completed
[INFO]:    Building libffi for armeabi-v7a
[INFO]:    libffi said it is already built, skipping
[INFO]:    Building python3 for armeabi-v7a
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/python3-libffi/armeabi-v7a__ndk_target_21/python3/android-build
[WARNING]: Doing some hacky stuff to link properly
[INFO]:    -> running cp /opt/android/android-ndk-r17c/platforms/android-21/arch-arm/usr/lib/crtbegin_so.o ./
[INFO]:    -> running cp /opt/android/android-ndk-r17c/platforms/android-21/arch-arm/usr/lib/crtend_so.o ./
[INFO]:    Activating flags for libffi
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Building setuptools for armeabi-v7a
[INFO]:    setuptools apparently isn't already in site-packages
[INFO]:    Installing setuptools into site-packages
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/setuptools-python3/armeabi-v7a__ndk_target_21/setuptools
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/python-installs/bdisttest_python3_googlendk --install-lib=.
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build --install-lib=Lib/site-packages
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Building pycparser for armeabi-v7a
[INFO]:    pycparser apparently isn't already in site-packages
[INFO]:    Installing pycparser into site-packages
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/pycparser-python3/armeabi-v7a__ndk_target_21/pycparser
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/python-installs/bdisttest_python3_googlendk --install-lib=.
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build --install-lib=Lib/site-packages
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Building cffi for armeabi-v7a
[INFO]:    cffi apparently isn't already in site-packages
[INFO]:    Building compiled components in cffi
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    -> running python setup.py clean --all
[INFO]:    -> running python setup.py build_ext -v
[INFO]:    -> running find build/lib.linux-x86_64-3.7 -name "*.o" -exec arm-linux-androideabi-strip --strip-unneeded {} ;
[INFO]:    <- directory context /home/user/testapps
[INFO]:    Installing cffi into site-packages
[INFO]:    -> directory context /home/user/.local/share/python-for-android/build/other_builds/cffi-python3/armeabi-v7a__ndk_target_21/cffi
[INFO]:    -> running python setup.py install -O2 --root=/home/user/.local/share/python-for-android/build/python-installs/bdisttest_python3_googlendk --install-lib=.
[INFO]:    Rebuilding compiled components in cffi
[INFO]:    -> running python setup.py clean --all
[INFO]:    -> running python setup.py build_ext -v
           working: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/home/user/.local/share/python-for-android/build/other_builds/libffi/armeabi-v7a__ndk_target_21/libffi/include -I/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/Include -I/home/user/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o

                                                                                                                                           Traceback (most recent call last):
  File "setup_testapp_python3.py", line 28, in <module>
    package_data={'testapp': ['*.py', '*.png']}
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/user/pythonforandroid/bdistapk.py", line 80, in run
    main()
  File "/home/user/pythonforandroid/toolchain.py", line 1051, in main
    ToolchainCL()
  File "/home/user/pythonforandroid/toolchain.py", line 560, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/home/user/pythonforandroid/toolchain.py", line 151, in wrapper_func
    build_dist_from_args(ctx, dist, args)
  File "/home/user/pythonforandroid/toolchain.py", line 192, in build_dist_from_args
    build_recipes(build_order, python_modules, ctx)
  File "/home/user/pythonforandroid/build.py", line 566, in build_recipes
    recipe.build_arch(arch)
  File "/home/user/pythonforandroid/recipe.py", line 890, in build_arch
    self.install_python_package(arch)
  File "/home/user/pythonforandroid/recipe.py", line 863, in install_python_package
    self.install_hostpython_package(arch)
  File "/home/user/pythonforandroid/recipe.py", line 908, in install_hostpython_package
    self.rebuild_compiled_components(arch, env)
  File "/home/user/pythonforandroid/recipe.py", line 917, in rebuild_compiled_components
    *self.setup_extra_args)
  File "/home/user/pythonforandroid/logger.py", line 193, in shprint
    shorten_string(msg, msg_width), width=msg_width))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2018' in position 70: ordinal not in range(128)
```

The error was also seen on Travis:
https://travis-ci.org/kivy/python-for-android/jobs/487456674


Discussion:

Resolving patch:
diff --git a/ci/constants.py b/ci/constants.py
--- a/ci/constants.py
+++ b/ci/constants.py
@@ -68,14 +68,15 @@ class TargetPython(Enum):
     'enum34',
     # https://github.com/kivy/python-for-android/issues/1399
     'libglob',
-    # cannot find -lcrystax
-    'cffi', 'pycryptodome', 'pymuk', 'secp256k1',
+    # Could not fetch URL https://pypi.org/simple/pymuk/: 404 Client Error
+    'pymuk',
+    # build_dir = glob.glob('build/lib.*')[0]
+    # IndexError: list index out of range
+    'secp256k1',
     # https://github.com/kivy/python-for-android/issues/1404
     'cryptography',
     # https://github.com/kivy/python-for-android/issues/1294
     'ffmpeg', 'ffpyplayer',
-    # https://github.com/kivy/python-for-android/pull/1307 ?
-    'gevent',
     'icu',
     # https://github.com/kivy/python-for-android/issues/1354
     'kivent_core', 'kivent_cymunk', 'kivent_particles', 'kivent_polygen',
diff --git a/pythonforandroid/recipes/gevent/__init__.py b/pythonforandroid/recipes/gevent/__init__.py
--- a/pythonforandroid/recipes/gevent/__init__.py
+++ b/pythonforandroid/recipes/gevent/__init__.py
@@ -4,7 +4,7 @@
 
 
 class GeventRecipe(CythonRecipe):
-    version = '1.3.7'
+    version = '1.4.0'
     url = 'https://pypi.python.org/packages/source/g/gevent/gevent-{version}.tar.gz'
     depends = ['librt', 'greenlet']
     patches = ["cross_compiling.patch"]
diff --git a/pythonforandroid/recipes/pycryptodome/__init__.py b/pythonforandroid/recipes/pycryptodome/__init__.py
--- a/pythonforandroid/recipes/pycryptodome/__init__.py
+++ b/pythonforandroid/recipes/pycryptodome/__init__.py
@@ -2,7 +2,7 @@
 
 
 class PycryptodomeRecipe(PythonRecipe):
-    version = '3.4.6'
+    version = '3.7.3'
     url = 'https://github.com/Legrandin/pycryptodome/archive/v{version}.tar.gz'
     depends = ['setuptools', 'cffi']
 


Test patch:
diff --git a/tests/test_logger.py b/tests/test_logger.py
new file mode 100644
--- /dev/null
+++ b/tests/test_logger.py
@@ -0,0 +1,18 @@
+import unittest
+from mock import MagicMock
+from pythonforandroid import logger
+
+
+class TestShprint(unittest.TestCase):
+
+    def test_unicode_encode(self):
+        """
+        Makes sure `shprint()` can handle unicode command output.
+        Running the test with PYTHONIOENCODING=ASCII env would fail, refs:
+        https://github.com/kivy/python-for-android/issues/1654
+        """
+        expected_command_output = ["foo\xa0bar"]
+        command = MagicMock()
+        command.return_value = expected_command_output
+        output = logger.shprint(command, 'a1', k1='k1')
+        self.assertEqual(output, expected_command_output)