Repository: tensorflow/tfx

Issue:
Import errors when trying to run Chicago Taxi on Dataflow
Similarly as in issue [#47](https://github.com/tensorflow/tfx/issues/47), I still have a problem with running CTE on Dataflow. When I use the code with no modifications, the error from previous issue persists - it seems that somehow the `try-except` around the imports doesn't do its job.

When I changed the code to include only the relative import in my fork [here](https://github.com/mwalenia/tfx/tree/import-fix), the problem disappeared, but another one manifested.

This time, there's a problem with importing `estimator` from tensorflow somewhere in the dependencies. Stacktrace:

```Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 773, in run
    self._load_main_session(self.local_staging_directory)
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 489, in _load_main_session
    pickler.load_session(session_file)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 269, in load_session
    return dill.load_session(file_path)
  File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 410, in load_session
    module = unpickler.load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 828, in _import_module
    return getattr(__import__(module, None, None, [obj]), obj)
  File "/usr/local/lib/python2.7/dist-packages/trainer/taxi.py", line 19, in <module>
    from tensorflow_transform import coders as tft_coders
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/__init__.py", line 19, in <module>
    from tensorflow_transform.analyzers import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/analyzers.py", line 39, in <module>
    from tensorflow_transform import tf_utils
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/tf_utils.py", line 24, in <module>
    from tensorflow.contrib.proto.python.ops import encode_proto_op
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/__init__.py", line 48, in <module>
    from tensorflow.contrib import distribute
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/distribute/__init__.py", line 34, in <module>
    from tensorflow.contrib.distribute.python.tpu_strategy import TPUStrategy
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/distribute/python/tpu_strategy.py", line 27, in <module>
    from tensorflow.contrib.tpu.python.ops import tpu_ops
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/__init__.py", line 73, in <module>
    from tensorflow.contrib.tpu.python.tpu.keras_support import tpu_model as keras_to_tpu_model
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/keras_support.py", line 71, in <module>
    from tensorflow.python.estimator import model_fn as model_fn_lib
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/__init__.py", line 25, in <module>
    import tensorflow.python.estimator.estimator_lib
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator_lib.py", line 22, in <module>
    from tensorflow.python.estimator.canned.baseline import BaselineClassifier
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/canned/baseline.py", line 50, in <module>
    from tensorflow.python.estimator import estimator
ImportError: cannot import name estimator
```

Is there anything I can do to fix this? 


Discussion:
@zhitaoli do you have any ideas as to why this happened?
This is likely a problem between our dependency and the Dataflow python SDK
container.
<https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies#beam-2x-python>

Are you using TFX 0.12.0, 0.13.0rc, or a version from GitHub master? Can
you share the pipeline side code (especially beam args)?

On Wed, May 8, 2019 at 4:23 AM Michal Walenia <notifications@github.com>
wrote:

> @zhitaoli <https://github.com/zhitaoli> do you have any ideas as to why
> this happened?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/tensorflow/tfx/issues/98#issuecomment-490448714>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAY6AZTALUOY7KWCFQPGWJDPUKZZDANCNFSM4HLQZK6Q>
> .
>


-- 
Cheers,

Zhitao Li

I got the same error using the setup.py libraries:

```
import setuptools

# LINT.IfChange
TF_VERSION = '1.12.0'
# LINT.ThenChange(train_mlengine.sh, start_model_server_mlengine.sh)

# LINT.IfChange
BEAM_VERSION = '2.11.0'
# LINT.ThenChange(setup_beam_on_flink.sh)

if __name__ == '__main__':
  setuptools.setup(
      name='tfx_chicago_taxi',
      version='0.12.0',
      packages=setuptools.find_packages(),
      install_requires=[
          'apache-beam[gcp]==' + BEAM_VERSION,
          'jupyter==1.0',
          'numpy==1.14.5',
          'protobuf==3.6.1',
          'tensorflow==' + TF_VERSION,
          'tensorflow-data-validation==0.12.0',
          'tensorflow-metadata==0.12.1',
          'tensorflow-model-analysis==0.12.1',
          'tensorflow-serving-api==1.12.0',
          'tensorflow-transform==0.12.0',
      ],
      python_requires='>=2.7,<3')
```
```
Traceback (most recent call last):
  File "tfdv_analyze_and_validate.py", line 191, in <module>
    main()
  File "tfdv_analyze_and_validate.py", line 176, in main
    pipeline_args=pipeline_args)
  File "tfdv_analyze_and_validate.py", line 121, in compute_stats
    statistics_pb2.DatasetFeatureStatisticsList)))
  File "/Users/gogasca/Documents/Development/dpe/tfx_taxi_cloud/taxi/lib/python2.7/site-packages/apache_beam/pipeline.py", line 426, in __exit__
    self.run().wait_until_finish()
  File "/Users/gogasca/Documents/Development/dpe/tfx_taxi_cloud/taxi/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1238, in wait_until_finish
    (self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 773, in run
    self._load_main_session(self.local_staging_directory)
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 489, in _load_main_session
    pickler.load_session(session_file)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 269, in load_session
    return dill.load_session(file_path)
  File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 410, in load_session
    module = unpickler.load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 828, in _import_module
    return getattr(__import__(module, None, None, [obj]), obj)
  File "/usr/local/lib/python2.7/dist-packages/trainer/taxi.py", line 19, in <module>
    from tensorflow_transform import coders as tft_coders
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/__init__.py", line 19, in <module>
    from tensorflow_transform.analyzers import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/analyzers.py", line 39, in <module>
    from tensorflow_transform import tf_utils
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_transform/tf_utils.py", line 24, in <module>
    from tensorflow.contrib.proto.python.ops import encode_proto_op
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/__init__.py", line 48, in <module>
    from tensorflow.contrib import distribute
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/distribute/__init__.py", line 34, in <module>
    from tensorflow.contrib.distribute.python.tpu_strategy import TPUStrategy
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/distribute/python/tpu_strategy.py", line 27, in <module>
    from tensorflow.contrib.tpu.python.ops import tpu_ops
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/__init__.py", line 73, in <module>
    from tensorflow.contrib.tpu.python.tpu.keras_support import tpu_model as keras_to_tpu_model
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/keras_support.py", line 71, in <module>
    from tensorflow.python.estimator import model_fn as model_fn_lib
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/__init__.py", line 25, in <module>
    import tensorflow.python.estimator.estimator_lib
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator_lib.py", line 22, in <module>
    from tensorflow.python.estimator.canned.baseline import BaselineClassifier
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/canned/baseline.py", line 50, in <module>
    from tensorflow.python.estimator import estimator
ImportError: cannot import name estimator
```
This is when running latest tfdv_analyze_and_validate_dataflow.sh
@zhitaoli I'm using the github master branch version. As for the pipeline, I'm using the one constructed in tfdv_analyze_and_validate_dataflow.sh and running the file with the command:

```
python tfdv_analyze_and_validate.py \
  --input bigquery-public-data.chicago_taxi_trips.taxi_trips \
  --infer_schema \
  --stats_path ${TFDV_OUTPUT_PATH}/train_stats.tfrecord \
  --schema_path ${SCHEMA_PATH} \
  --project ${MYPROJECT} \
  --region us-central1 \
  --temp_location ${TEMP_PATH} \
  --experiments shuffle_mode=auto \
  --job_name ${JOB_ID} \
  --setup_file ./setup.py \
  --save_main_session \
  --runner DataflowRunner \
  --max_rows=${MAX_ROWS}
```

Using:

```
import setuptools

# LINT.IfChange
TF_VERSION = '1.13.1'
# LINT.ThenChange(train_mlengine.sh, start_model_server_mlengine.sh)

# LINT.IfChange
BEAM_VERSION = '2.12.0'
# LINT.ThenChange(setup_beam_on_flink.sh)

if __name__ == '__main__':
  setuptools.setup(
      name='tfx_chicago_taxi',
      version='0.13.0',
      packages=setuptools.find_packages(),
      install_requires=[
          'apache-beam[gcp]==' + BEAM_VERSION,
          'jupyter==1.0',
          'protobuf==3.7.1',
          'tensorflow==' + TF_VERSION,
          'tensorflow-data-validation==0.13.1',
          'tensorflow-metadata==0.13.0',
          'tensorflow-model-analysis==0.13.2',
          'tensorflow-serving-api==1.13.0',
          'tensorflow-transform==0.13.0',
      ],
      python_requires='>=2.7,<3')
```
In a Python 2.7 virtualenv seems to be working now
@gogasca It worked for me too, thanks! What do we do now? Do you want to submit a PR updating the dependencies for chicago_taxi, or should I do it?

Once again, great find
There should be a submit from @ruoyu90 soon which will fix this. Thanks for the help.
This PR seems to include changes as well https://github.com/tensorflow/tfx/pull/91
> This PR seems to include changes as well #91

Yes this is the PR I mentioned. Waiting for @ruoyu90 to submit.
Thanks, just started training and looks like just some minor updates in chicago_taxi_client.py, task.py and model.py [needed](https://github.com/tensorflow/tfx/pull/91#issuecomment-491062271) in order to avoid import errors. 
@ruoyu90 It is necessary to limit the version of scikit-learn to 0.20 in setup.py of the chicago_taxi (and probably in more places), as version 0.21.0 requires python 3.5 since today
[source](https://scikit-learn.org/stable/whats_new.html#version-0-21)
@zhitaoli I also seem to have a problem with the `try-except` construct on Dataflow - even if it's included, my Dataflow runner fails with the same ImportError. Do you have any idea why can this happen?
@mwalenia thanks for the notice! We are close to release 0.13 which will have Python 3.5 support.
@ruoyu90 Hi, when can I expect a fix for this issue? #91 seems to be stagnant, should I submit my own PR?

Resolving patch:
diff --git a/tfx/examples/chicago_taxi/setup.py b/tfx/examples/chicago_taxi/setup.py
--- a/tfx/examples/chicago_taxi/setup.py
+++ b/tfx/examples/chicago_taxi/setup.py
@@ -15,28 +15,29 @@
 import setuptools
 
 # LINT.IfChange
-TF_VERSION = '1.12.0'
+TF_VERSION = '1.13.1'
 # LINT.ThenChange(train_mlengine.sh, start_model_server_mlengine.sh)
 
 # LINT.IfChange
-BEAM_VERSION = '2.11.0'
+BEAM_VERSION = '2.12.0'
 # LINT.ThenChange(setup_beam_on_flink.sh)
 
 if __name__ == '__main__':
   setuptools.setup(
       name='tfx_chicago_taxi',
-      version='0.12.0',
+      version='0.13.0',
       packages=setuptools.find_packages(),
       install_requires=[
-          'apache-beam[gcp]==' + BEAM_VERSION,
-          'jupyter==1.0',
-          'numpy==1.14.5',
-          'protobuf==3.6.1',
-          'tensorflow==' + TF_VERSION,
-          'tensorflow-data-validation==0.12.0',
-          'tensorflow-metadata==0.12.1',
-          'tensorflow-model-analysis==0.12.1',
-          'tensorflow-serving-api==1.12.0',
-          'tensorflow-transform==0.12.0',
+          'apache-beam[gcp]>=' + BEAM_VERSION,
+          'jupyter>=1.0,<2',
+          'notebook>=5.7.8,<5.8',
+          'numpy>=1.14.5,<2',
+          'protobuf>=3.7.0,<3.8.0',
+          'tensorflow>=' + TF_VERSION,
+          'tensorflow-data-validation>=0.13.1,<0.14',
+          'tensorflow-metadata>=0.13.1,<0.14',
+          'tensorflow-model-analysis>=0.13.2,<0.14',
+          'tensorflow-serving-api>=1.13.0,<1.14',
+          'tensorflow-transform>=0.13.0,<0.14',
       ],
-      python_requires='>=2.7,<3')
+      python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<4',)


Test patch: