Repository: pantsbuild/pants

Issue:
Tensorflow used in example project fails to compile with Python 3.7
When running `PY=python3.7  ./pants test examples/tests/python/example_test/tensorflow_custom_op:tensorflow_custom_op`, we get the exception `undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs` https://travis-ci.org/pantsbuild/pants/jobs/509277221#L2171

### Potential cause: gcc flags

I suspected this is because we aren’t compiling with the flags TensorFlow says you need to use
```TF_CFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') )
TF_LFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') )
g++ -std=c++11 -shared zero_out.cc -o zero_out.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2
```

Stack Overflow suggests this is the issue https://stackoverflow.com/a/49886418, and TF says to do these steps when compiling https://www.tensorflow.org/guide/extend/op#top_of_page

@cosmicexplorer explained this is likely not the issue for these reasons:
(1) we link with `-ltensorflow_framework`
(2) `compiler_option_sets` can be used for this sort of thing, but that shouldn't be the solution to this

### Potential cause: gcc version used changing?
E.g. the wheel used to build the Py37 wheel is different and causing some issue.


Discussion:
I repro this and am looking into it.
Note that https://www.tensorflow.org/guide/extend/op also says to use `self.test_session()`, which is deprecated, so there may be more surprises we get to scrape the tensorflow changelog for.
Ugh, it's literally just the `-D_GLIBCXX_USE_CXX11_ABI=0` line is out of date on the master version of https://www.tensorflow.org/guide/extend/op. It's supposed to be `-D_GLIBCXX_USE_CXX11_ABI=1` (as in, it works in python 3.7 if you set it to that in `pants.ini`). Thinking now about how we might want to set different flags for different python versions.

Resolving patch:
diff --git a/src/python/pants/backend/native/tasks/native_compile.py b/src/python/pants/backend/native/tasks/native_compile.py
--- a/src/python/pants/backend/native/tasks/native_compile.py
+++ b/src/python/pants/backend/native/tasks/native_compile.py
@@ -167,6 +167,7 @@ def _make_compile_request(self, versioned_target):
     sources_and_headers = self.get_sources_headers_for_target(target)
     compiler_option_sets = (self._compile_settings.native_build_step
                                 .get_compiler_option_sets_for_target(target))
+    self.context.log.debug('target: {}, compiler_option_sets: {}'.format(target, compiler_option_sets))
 
     compile_request = NativeCompileRequest(
       compiler=self._compiler(target),
diff --git a/src/python/pants/option/compiler_option_sets_mixin.py b/src/python/pants/option/compiler_option_sets_mixin.py
--- a/src/python/pants/option/compiler_option_sets_mixin.py
+++ b/src/python/pants/option/compiler_option_sets_mixin.py
@@ -4,11 +4,15 @@
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+import logging
 from builtins import object
 
 from pants.util.meta import classproperty
 
 
+logger = logging.getLogger(__name__)
+
+
 class CompilerOptionSetsMixin(object):
   """A mixin for language-scoped that support compiler option sets."""
 
@@ -21,6 +25,9 @@ def register_options(cls, register):
     register('--compiler-option-sets-disabled-args', advanced=True, type=dict, fingerprint=True,
              default=cls.get_compiler_option_sets_disabled_default_value,
              help='Extra compiler args to use for each disabled option set.')
+    register('--default-compiler-option-sets', advanced=True, type=list, fingerprint=True,
+             default=cls.get_default_compiler_option_sets,
+             help="The compiler_option_sets to use for targets which don't declare any.")
 
   @classproperty
   def get_compiler_option_sets_enabled_default_value(cls):
@@ -32,6 +39,11 @@ def get_compiler_option_sets_disabled_default_value(cls):
     """Override to set default for this option."""
     return {}
 
+  @classproperty
+  def get_default_compiler_option_sets(cls):
+    """Override to set the default compiler_option_sets for targets which don't declare them."""
+    return []
+
   @classproperty
   def get_fatal_warnings_enabled_args_default(cls):
     """Override to set default for this option."""
@@ -45,6 +57,11 @@ def get_fatal_warnings_disabled_args_default(cls):
   # TODO(mateo): The compiler_option_sets could use an API that requires implementing platforms to
   # surface documentation - this took me awhile to unwind.
   def get_merged_args_for_compiler_option_sets(self, compiler_option_sets):
+    # If no option sets are provided, use the default value.
+    if not compiler_option_sets:
+      compiler_option_sets = self.get_options().default_compiler_option_sets
+      logger.debug('using default option sets: {}'.format(compiler_option_sets))
+
     compiler_options = set()
 
     # Set values for enabled options.


Test patch:
diff --git a/examples/tests/python/example_test/tensorflow_custom_op/test_zero_out_op.py b/examples/tests/python/example_test/tensorflow_custom_op/test_zero_out_op.py
--- a/examples/tests/python/example_test/tensorflow_custom_op/test_zero_out_op.py
+++ b/examples/tests/python/example_test/tensorflow_custom_op/test_zero_out_op.py
@@ -4,9 +4,6 @@
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-import sys
-import unittest
-
 import tensorflow as tf
 
 from example.tensorflow_custom_op.zero_out_custom_op import zero_out_module
@@ -15,8 +12,7 @@
 # This code is from the guide in https://www.tensorflow.org/guide/extend/op.
 class ZeroOutTest(tf.test.TestCase):
 
-  @unittest.skipIf(sys.version_info[0:2] == (3, 7), "See https://github.com/pantsbuild/pants/issues/7417.")
   def test_zero_out(self):
-    with self.test_session():
+    with self.cached_session():
       result = zero_out_module().zero_out([5, 4, 3, 2, 1])
       self.assertAllEqual(result.eval(), [5, 0, 0, 0, 0])