Repository: bazelbuild/bazel
Issue: Make android tools scripts Python3-compatible for building android apps
Labels: type: feature request

### Description of the problem / feature request / question:

Google's TFLite team has discovered that the TF Lite Android demo application cannot build via Bazel with Python 3, because the android tools Bazel uses aren't Python-3 compatible. The first breaking point looks like `third_party/gflags`, but past that there seem to be additional version incompatibilities.

### If possible, provide a minimal example to reproduce the problem:

Here's a very fast example:
```shell
$ git clone http://github.com/bazelbuild/bazel /tmp/bazel
$ cd /tmp/bazel
$ python3
>>> import tools.android.aar_embedded_jars_extractor
```
I'm not actually sure how to do the same as the above in python 2, but that sequence of commands results in the same errors we see when trying to build the demo app (`No module named 'gflags'`). I can sort of fix this with `$ pip install python-gflags`, which makes Python not even try to use the local files (this seems like an ultimately buggy workaround). The import is fixed, but the build (below) is still busted with a str-to-buffer message problem which I've included at the very bottom of the issue.

Here's how I ran the TF Lite app build. The build requires the [SDK setup steps for Tensorflow's WORKSPACE](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/java/demo).

```shell
$ virtualenv --python=python3 --system-site-packages /tmp/py3
$ git clone http://github.com/tensorflow/tensorflow /tmp/tf
$ cd /tmp/tf
$ source /tmp/py3/bin/activate
# prepare WORKSPACE, see link above
$ ./configure      # should default to /tmp/py3 python path
$ bazel build //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo --cxxopt='--std=c++11' --config=android_arm 
```

### Environment info

* Operating System: Ubuntu 14.04 LTS
* Bazel version (output of `bazel info release`): release 0.7.0

###  Have you found anything relevant by searching the web?

- #1580 seems generally relevant.
- [A relevant-looking issue from rules_docker](https://github.com/bazelbuild/rules_docker/issues/169)
- [Building the tflite demo app from source](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/java/demo)
- [Where installing python-gflags was noted](https://github.com/mrmeku/brackets-closure-linter/issues/3)
- [An SO question on the str buffer error](https://stackoverflow.com/questions/5471158/typeerror-str-does-not-support-the-buffer-interface)

### Anything else, information or logs or outputs that would be helpful?

This incompatibility means that the TF Lite app can't be built (afaik) unless users install Python 2. Not a tremendous problem, but pretty inconvenient.

From the build commands above, I get something like this:

```
(py3)❯ bazel build //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo --cxxopt='--std=c++11' --config=android_arm --logging=0
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 15. The major revisions supported by Bazel are [10, 11, 12, 13, 14]. Defaulting to revision 14.
INFO: Analysed target //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo (0 packages loaded).
INFO: Found 1 target...
ERROR: /usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/external/androidsdk/com.android.support/BUILD:4563:1: Filtering AAR native libs by architecture failed (Exit 1)
Traceback (most recent call last):
  File "/usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/execroot/org_tensorflow/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/org_tensorflow/../bazel_tools/tools/android/aar_native_libs_zip_creator.py", line 26, in <module>
    from third_party.py import gflags
  File "/usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/execroot/org_tensorflow/bazel-out/host/bin/external/bazel_tools/tools/android/aar_native_libs_zip_creator.runfiles/bazel_tools/third_party/py/gflags/__init__.py", line 1, in <module>
    from gflags import *
ImportError: No module named 'gflags'
Target //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.440s, Critical Path: 0.11s
FAILED: Build did NOT complete successfully
```
And here's the str-to-buffer problem after installing `pip install python-gflags`:

```
(py3)❯ bazel build //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo --cxxopt='--std=c++11' --config=android_arm --logging=0
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 15. The major revisions supported by Bazel are [10, 11, 12, 13, 14]. Defaulting to revision 14.
INFO: Analysed target //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo (50 packages loaded).
INFO: Found 1 target...
ERROR: /usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/external/androidsdk/com.android.support/BUILD:4563:1: Extracting classes.jar and libs/*.jar from support-fragment-25.2.0.aar failed (Exit 1)
Traceback (most recent call last):
  File "/usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/execroot/org_tensorflow/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/org_tensorflow/../bazel_tools/tools/android/aar_embedded_jars_extractor.py", line 92, in <module>
    main()
  File "/usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/execroot/org_tensorflow/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/org_tensorflow/../bazel_tools/tools/android/aar_embedded_jars_extractor.py", line 87, in main
    _Main(FLAGS.input_aar, FLAGS.output_singlejar_param_file, FLAGS.output_dir)
  File "/usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/execroot/org_tensorflow/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/org_tensorflow/../bazel_tools/tools/android/aar_embedded_jars_extractor.py", line 67, in _Main
    output_dir_orig)
  File "/usr/local/google/home/angerson/.cache/bazel/_bazel_angerson/7b7f16644e8d0eb6d37d6401f95d6305/execroot/org_tensorflow/bazel-out/host/bin/external/bazel_tools/tools/android/aar_embedded_jars_extractor.runfiles/org_tensorflow/../bazel_tools/tools/android/aar_embedded_jars_extractor.py", line 48, in ExtractEmbeddedJars
    singlejar_param_file.write("--exclude_build_data\n")
TypeError: 'str' does not support the buffer interface
Target //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 10.214s, Critical Path: 0.54s
FAILED: Build did NOT complete successfully
```

Issue comments:
I think that #3855 should resolve this issue, but I'll test it to confirm.
Hey thanks for bringing this to our attention, ccing some people that can help
@jin @dkelmer @ahumesky 
Thanks @aj-michael - I was able to reproduce @angersson's errors and can confirm that #3855 resolves the first one.

However, the second error persists. It can probably be resolved by updating modules in `tools/android` and `tools/build_defs` for compatibility with python3-style strings. Should be relatively simple, so I can probably make those changes soon.
As promised, I've created a branch containing [python 3 compatibility updates](https://github.com/bazelbuild/bazel/compare/master...akira-baruah:python3-tools) to bazel tools.

@angersson @aj-michael @jin @dkelmer @ahumesky you can test the fix by running:
```
git clone https://github.com/akira-baruah/bazel.git /tmp/bazel-akira
cd /tmp/bazel-akira
git checkout python3-tools  # branch with bugfixes
bazel build //src:bazel

cd <tensorflow repo>
/tmp/bazel-akira/bazel-bin/src/bazel build //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo --cxxopt='--std=c++11' --force_python=py3 --python_path=$(which python3) --config=android_arm --logging=0
```
If it looks good to you, I'll make a PR. Please note that this branch depends on PR #3855, so we would probably want to have that merged first.

Thanks!
I was able to replicate the fixes, but I'll need to fetch NDK 15 to fully test building the app in a clean environment due to #4068.
I verified build success with nvidia-docker, `gcr.io/tensorflow/tensorflow:latest-devel-gpu`, and this... awful script, run inside the container:

```shell
#!/usr/bin/env bash

set -euxo pipefail

TMPDIR=$(mktemp -d)
trap '' PIPE

git clone https://github.com/akira-baruah/bazel.git ${TMPDIR}/bazel-akira
cd ${TMPDIR}/bazel-akira
git checkout python3-tools  # branch with bugfixes
bazel build //src:bazel

cd ${TMPDIR}

curl -o ${TMPDIR}/sdk-tools.zip https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip
unzip ${TMPDIR}/sdk-tools.zip -d ${TMPDIR}/sdk
echo "y" | ${TMPDIR}/sdk/tools/bin/sdkmanager "build-tools;26.0.1" "platforms;android-26" "extras;android;m2repository"

pip install virtualenv
virtualenv --python=python3 --system-site-packages ${TMPDIR}/py3
PS1=""  # Avoids `set -u`-related error with bin/activate
source ${TMPDIR}/py3/bin/activate

git clone http://github.com/tensorflow/tensorflow ${TMPDIR}/tf
cd ${TMPDIR}/tf

curl -o ${TMPDIR}/ndk.zip https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip
unzip ${TMPDIR}/ndk.zip -d ${TMPDIR}/ndk/

cat >> ${TMPDIR}/tf/WORKSPACE <<HEREDOC
android_sdk_repository(
    name = "androidsdk",
    api_level = 26,
    build_tools_version = "26.0.1",
    path = "${TMPDIR}/sdk",
)
android_ndk_repository(
    name="androidndk",
    path="${TMPDIR}/ndk/android-ndk-r14b",
    api_level=14)
HEREDOC

set +o pipefail
yes '' | ${TMPDIR}/tf/configure
set -o pipefail
cd ${TMPDIR}/tf
${TMPDIR}/bazel-akira/bazel-bin/src/bazel build //tensorflow/contrib/lite/java/demo/app/src/main:TfLiteCameraDemo --cxxopt='--std=c++11' --config=android_arm
```

Sweet.
@angersson now that gflags supports python 3 imports (see commits fb15f0f4e4885243d354a14997d09463594385bf, d926bc40260549b997a6a5a1e82d9e7999dbb65e, and 4d0e6265911199b1376d0f52e249625180a0500d), I've created PR #4265 to resolve this issue.

Pull request review comments:

Patch that resolved the issue:
diff --git a/tools/android/aar_embedded_jars_extractor.py b/tools/android/aar_embedded_jars_extractor.py
index a46dd43b6fa538..393d3f4c5c3732 100644
--- a/tools/android/aar_embedded_jars_extractor.py
+++ b/tools/android/aar_embedded_jars_extractor.py
@@ -62,7 +62,7 @@ def _Main(input_aar,
   if not output_dir_orig:
     output_dir_orig = output_dir
   with zipfile.ZipFile(input_aar, "r") as aar:
-    with open(output_singlejar_param_file, "wb") as singlejar_param_file:
+    with open(output_singlejar_param_file, "w") as singlejar_param_file:
       ExtractEmbeddedJars(aar, singlejar_param_file, output_dir,
                           output_dir_orig)
 
diff --git a/tools/android/aar_native_libs_zip_creator.py b/tools/android/aar_native_libs_zip_creator.py
index 0195d01cadb115..bcf5931ec7072d 100644
--- a/tools/android/aar_native_libs_zip_creator.py
+++ b/tools/android/aar_native_libs_zip_creator.py
@@ -63,8 +63,8 @@ def Main(input_aar_path, output_zip_path, cpu, input_aar_path_for_error_msg):
       try:
         CreateNativeLibsZip(input_aar, cpu, native_libs_zip)
       except UnsupportedArchitectureException:
-        print("AAR " + input_aar_path_for_error_msg +
-              " missing native libs for requested architecture: " + cpu)
+        print(("AAR " + input_aar_path_for_error_msg +
+              " missing native libs for requested architecture: " + cpu))
         sys.exit(1)
 
 
diff --git a/tools/android/merge_manifests.py b/tools/android/merge_manifests.py
index 880c55f8503b12..3637030a990d7e 100644
--- a/tools/android/merge_manifests.py
+++ b/tools/android/merge_manifests.py
@@ -449,7 +449,7 @@ def main():
   if FLAGS.exclude_permission:
     warning = _ValidateAndWarnPermissions(FLAGS.exclude_permission)
     if warning:
-      print warning
+      print(warning)
 
   merged_manifests = MergeManifests(_ReadFile(FLAGS.merger),
                                     _ReadFiles(FLAGS.mergee),
diff --git a/tools/android/merge_manifests_test.py b/tools/android/merge_manifests_test.py
index 62d0474e15a065..7567a489f7a48f 100644
--- a/tools/android/merge_manifests_test.py
+++ b/tools/android/merge_manifests_test.py
@@ -532,12 +532,12 @@ def testMerge(self):
         ['android.permission.READ_LOGS'])
     result = merger.Merge()
     expected = xml.dom.minidom.parseString(MANUALLY_MERGED).toprettyxml()
-    self.assertEquals(Reformat(expected), Reformat(result))
+    self.assertEqual(Reformat(expected), Reformat(result))
 
   def testReformat(self):
     text = '  a\n  b\n\n\n \t c'
     expected = 'a\nb\nc'
-    self.assertEquals(expected, Reformat(text))
+    self.assertEqual(expected, Reformat(text))
 
   def testValidateAndWarnPermissions(self):
     permissions = ['android.permission.VIBRATE', 'android.permission.LAUGH']
@@ -589,7 +589,7 @@ def testMergeToCreateValidManifest(self):
         ['all'])
     result = merger.Merge()
     expected = xml.dom.minidom.parseString(VALID_MANIFEST).toprettyxml()
-    self.assertEquals(Reformat(expected), Reformat(result))
+    self.assertEqual(Reformat(expected), Reformat(result))
 
   def testMergeWithNoApplication(self):
     merger = merge_manifests.MergeManifests(
@@ -609,7 +609,7 @@ def testMergeWithNamespaces(self):
         MERGED_MANIFEST_WITH_EXTRA_NAMESPACE).toprettyxml()
     # Make sure the result is valid xml (not missing xmlns declarations)
     result_reparsed = xml.dom.minidom.parseString(result).toprettyxml()
-    self.assertEquals(Reformat(expected), Reformat(result_reparsed))
+    self.assertEqual(Reformat(expected), Reformat(result_reparsed))
 
   def testMergeConflictingNamespaces(self):
     self.maxDiff = None
@@ -618,7 +618,7 @@ def testMergeConflictingNamespaces(self):
          'MANIFEST_WITH_CONFLICTING_NAMESPACE'),
         [(MANIFEST_WITH_EXTRA_NAMESPACE, 'MANIFEST_WITH_EXTRA_NAMESPACE')],
         ['all'])
-    with self.assertRaisesRegexp(merge_manifests.MalformedManifestException,
+    with self.assertRaisesRegex(merge_manifests.MalformedManifestException,
                                  'different values for namespace xmlns:tools'):
       merger.Merge()
 
diff --git a/tools/android/resource_extractor.py b/tools/android/resource_extractor.py
index 2cd11d907c2809..473398cd4cc99c 100644
--- a/tools/android/resource_extractor.py
+++ b/tools/android/resource_extractor.py
@@ -95,7 +95,7 @@ def ExtractResources(input_jar, output_zip):
 
 def main(argv):
   if len(argv) != 3:
-    print USAGE
+    print(USAGE)
     sys.exit(1)
   with zipfile.ZipFile(argv[1], 'r') as input_jar:
     with zipfile.ZipFile(argv[2], 'w') as output_zip:
diff --git a/tools/android/stubify_manifest.py b/tools/android/stubify_manifest.py
index f415774938cd88..52a5be9f9ff251 100644
--- a/tools/android/stubify_manifest.py
+++ b/tools/android/stubify_manifest.py
@@ -164,5 +164,5 @@ def main():
   try:
     main()
   except BadManifestException as e:
-    print e
+    print(e)
     sys.exit(1)
diff --git a/tools/build_defs/pkg/archive.py b/tools/build_defs/pkg/archive.py
index 9d5ddfa667d350..59728e0655b1cc 100644
--- a/tools/build_defs/pkg/archive.py
+++ b/tools/build_defs/pkg/archive.py
@@ -35,7 +35,7 @@ class SimpleArFile(object):
   with SimpleArFile(filename) as ar:
     nextFile = ar.next()
     while nextFile:
-      print nextFile.filename
+      print(nextFile.filename)
       nextFile = ar.next()
 
   Upon error, this class will raise a ArError exception.
diff --git a/tools/objc/protobuf_compiler.py b/tools/objc/protobuf_compiler.py
index bd789b7ddd0352..de6c4b9a702ba4 100644
--- a/tools/objc/protobuf_compiler.py
+++ b/tools/objc/protobuf_compiler.py
@@ -18,5 +18,5 @@
 import sys
 
 if __name__ == '__main__':
-  print 'Bazel does not yet support protobuf compiling.'
+  print('Bazel does not yet support protobuf compiling.')
   sys.exit(1)


Test patch: