Repository: ansible/ansible

Issue:
Error while using gcpubsub module
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
gcpubsub

##### ANSIBLE VERSION
```
2.3.0, devel
```

##### OS / ENVIRONMENT
N/A

##### SUMMARY
When trying to create Google Cloud Engine PubSub topic, I'm getting an error:
```
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_nLEh1e/ansible_module_gcpubsub.py", line 328, in <module>
    from ansible.module_utils.gcp import *
  File "/tmp/ansible_nLEh1e/ansible_modlib.zip/ansible/module_utils/gcp.py", line 64, in <module>
ImportError: No module named utils.display
```
Same on macOS Sierra and on Ubuntu 16.04. Tried to install by apt, pip and from source.

Tried to remove `utils.display` usage from `module_utils/gcp.py` and build from source, which fixed the error.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Run the example playbook:
<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- name: Pubsub example playbook
  hosts: "{{ target }}"
  tasks:
    - name: Create pubsub topic
      gcpubsub:
        topic: test_topic
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
PubSub topic is created.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
(IP address hidden)
```
TASK [Create pubsub topics] *****************************************************************************************************
task path: /Users/dizeee/srv/billie.io/provisioner/configure-sites.yml:16
Using module file /Library/Python/2.7/site-packages/ansible/modules/cloud/google/gcpubsub.py
<X.X.X.X> ESTABLISH SSH CONNECTION FOR USER: None
<X.X.X.X> SSH: EXEC ssh -F ./ssh.conf -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=20 -o 'ControlPath=~/.ssh/connection-ansible-%r@%h:%p' X.X.X.X '/bin/sh -c '"'"'sudo sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-cpcfnavcmckoogenmykvzsqwoabwjtzl; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<X.X.X.X> (1, '', 'Traceback (most recent call last):\n  File "/tmp/ansible_nLEh1e/ansible_module_gcpubsub.py", line 328, in <module>\n    from ansible.module_utils.gcp import *\n  File "/tmp/ansible_nLEh1e/ansible_modlib.zip/ansible/module_utils/gcp.py", line 64, in <module>\nImportError: No module named utils.display\n')
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_nLEh1e/ansible_module_gcpubsub.py", line 328, in <module>
    from ansible.module_utils.gcp import *
  File "/tmp/ansible_nLEh1e/ansible_modlib.zip/ansible/module_utils/gcp.py", line 64, in <module>
ImportError: No module named utils.display

fatal: [X.X.X.X]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_nLEh1e/ansible_module_gcpubsub.py\", line 328, in <module>\n    from ansible.module_utils.gcp import *\n  File \"/tmp/ansible_nLEh1e/ansible_modlib.zip/ansible/module_utils/gcp.py\", line 64, in <module>\nImportError: No module named utils.display\n",
    "module_stdout": "",
    "rc": 1
}

MSG:

MODULE FAILURE
```


Discussion:
cc @supertom
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
FWIW, I've hit this too, but not on the gcpubsub module.  @dizeee, I see you also commented on https://github.com/jlund/streisand/issues/629.

I haven't tracked down root cause yet though.
module_utils/gcp.py imports code from 'ansible.utils' which causes problems

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/gcp.py#L63-L68

Though I don't see it documented anywhere, that should be avoided, since the runtime for the module likely wont have the app code on the path ('ansible.utils' etc).
@dizeee @alikins @erjohnso, Yeah if your connections aren't local, then this will happen.  We only use display to print out the deprecation warnings, so there's no deep dependency on that ansible.utils.

Resolving patch:
diff --git a/lib/ansible/module_utils/gcp.py b/lib/ansible/module_utils/gcp.py
--- a/lib/ansible/module_utils/gcp.py
+++ b/lib/ansible/module_utils/gcp.py
@@ -60,13 +60,6 @@
     HAS_GOOGLE_API_LIB = False
 
 
-# Ansible Display object for warnings
-try:
-    from __main__ import display
-except ImportError:
-    from ansible.utils.display import Display
-    display = Display()
-
 import ansible.module_utils.six.moves.urllib.parse as urlparse
 
 GCP_DEFAULT_SCOPES = ['https://www.googleapis.com/auth/cloud-platform']
@@ -106,7 +99,7 @@ def _get_gcp_environment_credentials(service_account_email, credentials_file, pr
     return (service_account_email, credentials_file, project_id)
 
 
-def _get_gcp_libcloud_credentials(service_account_email=None, credentials_file=None, project_id=None):
+def _get_gcp_libcloud_credentials(module, service_account_email=None, credentials_file=None, project_id=None):
     """
     Helper to look for libcloud secrets.py file.
 
@@ -130,9 +123,9 @@ def _get_gcp_libcloud_credentials(service_account_email=None, credentials_file=N
     if service_account_email is None or credentials_file is None:
         try:
             import secrets
-            display.deprecated(msg=("secrets file found at '%s'.  This method of specifying "
-                                    "credentials is deprecated.  Please use env vars or "
-                                    "Ansible YAML files instead" % (secrets.__file__)), version=2.5)
+            module.deprecate(msg=("secrets file found at '%s'.  This method of specifying "
+                                  "credentials is deprecated.  Please use env vars or "
+                                  "Ansible YAML files instead" % (secrets.__file__)), version=2.5)
         except ImportError:
             secrets = None
         if hasattr(secrets, 'GCE_PARAMS'):
@@ -199,7 +192,7 @@ def _get_gcp_credentials(module, require_valid_json=True, check_libcloud=False):
     # get the remaining values from the libcloud secrets file.
     (service_account_email,
      credentials_file,
-     project_id) = _get_gcp_libcloud_credentials(service_account_email,
+     project_id) = _get_gcp_libcloud_credentials(module, service_account_email,
                                                  credentials_file, project_id)
 
     if credentials_file is None or project_id is None or service_account_email is None:
@@ -274,9 +267,9 @@ def _validate_credentials_file(module, credentials_file, require_valid_json=True
             module.fail_json(
                 msg='GCP Credentials File %s invalid.  Must be valid JSON.' % credentials_file, changed=False)
         else:
-            display.deprecated(msg=("Non-JSON credentials file provided. This format is deprecated. "
-                                    " Please generate a new JSON key from the Google Cloud console"),
-                               version=2.5)
+            module.deprecate(msg=("Non-JSON credentials file provided. This format is deprecated. "
+                                  " Please generate a new JSON key from the Google Cloud console"),
+                             version=2.5)
             return True
 
 


Test patch:
diff --git a/test/sanity/pep8/legacy-files.txt b/test/sanity/pep8/legacy-files.txt
--- a/test/sanity/pep8/legacy-files.txt
+++ b/test/sanity/pep8/legacy-files.txt
@@ -890,7 +890,6 @@ test/units/module_utils/basic/test_run_command.py
 test/units/module_utils/basic/test_safe_eval.py
 test/units/module_utils/basic/test_set_mode_if_different.py
 test/units/module_utils/ec2/test_aws.py
-test/units/module_utils/gcp/test_auth.py
 test/units/module_utils/json_utils/test_filter_non_json_lines.py
 test/units/module_utils/test_basic.py
 test/units/module_utils/test_distribution_version.py
diff --git a/test/units/module_utils/gcp/test_auth.py b/test/units/module_utils/gcp/test_auth.py
--- a/test/units/module_utils/gcp/test_auth.py
+++ b/test/units/module_utils/gcp/test_auth.py
@@ -27,6 +27,8 @@
 
 # Fake data/function used for testing
 fake_env_data = {'GCE_EMAIL': 'gce-email'}
+
+
 def fake_get_gcp_environ_var(var_name, default_value):
     if var_name not in fake_env_data:
         return default_value
@@ -34,6 +36,8 @@ def fake_get_gcp_environ_var(var_name, default_value):
         return fake_env_data[var_name]
 
 # Fake AnsibleModule for use in tests
+
+
 class FakeModule(object):
     class Params():
         data = {}
@@ -51,6 +55,10 @@ def __init__(self, data={}):
     def fail_json(self, **kwargs):
         raise ValueError("fail_json")
 
+    def deprecate(self, **kwargs):
+        return None
+
+
 class GCPAuthTestCase(unittest.TestCase):
     """Tests to verify different Auth mechanisms."""
 
@@ -74,28 +82,30 @@ def test_get_gcp_environ_var(self):
         existing_var_name = 'gcp_ansible_auth_test_54321'
         non_existing_var_name = 'doesnt_exist_gcp_ansible_auth_test_12345'
         os.environ[existing_var_name] = 'foobar'
-        self.assertEqual('foobar', _get_gcp_environ_var(existing_var_name, None))
+        self.assertEqual('foobar', _get_gcp_environ_var(
+            existing_var_name, None))
         del os.environ[existing_var_name]
         self.assertEqual('default_value', _get_gcp_environ_var(
             non_existing_var_name, 'default_value'))
 
     def test_get_gcp_libcloud_credentials_no_import(self):
         """No secrets imported.  Whatever is sent in should come out."""
-        actual = _get_gcp_libcloud_credentials(service_account_email=None,
+        module = FakeModule()
+        actual = _get_gcp_libcloud_credentials(module,
+                                               service_account_email=None,
                                                credentials_file=None,
                                                project_id=None)
         expected = (None, None, None)
         self.assertEqual(expected, actual)
         # no libcloud, with values
-        actual = _get_gcp_libcloud_credentials(service_account_email='sa-email',
+        actual = _get_gcp_libcloud_credentials(module,
+                                               service_account_email='sa-email',
                                                credentials_file='creds-file',
                                                project_id='proj-id')
         expected = ('sa-email', 'creds-file', 'proj-id')
         self.assertEqual(expected, actual)
 
-    @mock.patch("ansible.utils.display.Display.deprecated",
-                name='mock_deprecated', return_value=None)
-    def test_get_gcp_libcloud_credentials_import(self, mock_deprecated):
+    def test_get_gcp_libcloud_credentials_import(self):
         """secrets is imported and those values should be used."""
         # Note: Opted for a real class here rather than MagicMock as
         # __getitem__ comes for free.
@@ -109,21 +119,25 @@ def __init__(self):
 
         # patch in module
         fake_secrets = FakeSecrets()
-        patcher = mock.patch.dict(sys.modules,{'secrets': fake_secrets})
+        patcher = mock.patch.dict(sys.modules, {'secrets': fake_secrets})
         patcher.start()
 
         # obtain sa and creds from secrets
-        actual = _get_gcp_libcloud_credentials(service_account_email=None,
+        module = FakeModule()
+        actual = _get_gcp_libcloud_credentials(module,
+                                               service_account_email=None,
                                                credentials_file=None,
                                                project_id='proj-id')
         expected = ('secrets-sa', 'secrets-file.json', 'proj-id')
         self.assertEqual(expected, actual)
 
-        # fetch project id.  Current logic requires sa-email or creds to be set.
+        # fetch project id.  Current logic requires sa-email or creds to be
+        # set.
         fake_secrets.GCE_KEYWORD_PARAMS['project'] = 'new-proj-id'
         fake_secrets.GCE_PARAMS[1] = 'my-creds.json'
-
-        actual = _get_gcp_libcloud_credentials(service_account_email='my-sa',
+        module = FakeModule()
+        actual = _get_gcp_libcloud_credentials(module,
+                                               service_account_email='my-sa',
                                                credentials_file=None,
                                                project_id=None)
         expected = ('my-sa', 'my-creds.json', 'new-proj-id')
@@ -132,9 +146,7 @@ def __init__(self):
         # stop patching
         patcher.stop()
 
-    @mock.patch("ansible.utils.display.Display.deprecated",
-                name='mock_deprecated', return_value=None)
-    def test_validate_credentials_file(self, mock_deprecated):
+    def test_validate_credentials_file(self):
         # TODO(supertom): Only dealing with p12 here, check the other states
         # of this function
         module = mock.MagicMock()
@@ -190,7 +202,8 @@ def test_get_gcp_environment_credentials(self, mockobj):
         # data passed in, picking up project id only
         fake_env_data = {'GOOGLE_CLOUD_PROJECT': 'my-project'}
         expected = tuple(['my-sa-email', '/path/to/creds.json', 'my-project'])
-        actual = _get_gcp_environment_credentials('my-sa-email', '/path/to/creds.json', None)
+        actual = _get_gcp_environment_credentials(
+            'my-sa-email', '/path/to/creds.json', None)
         self.assertEqual(expected, actual)
 
     @mock.patch('ansible.module_utils.gcp._get_gcp_environ_var',
@@ -207,7 +220,8 @@ def test_get_gcp_credentials(self, mockobj):
 
         # project_id (only) is set from Ansible params.
         module.params.data['project_id'] = 'my-project'
-        actual = _get_gcp_credentials(module, require_valid_json=True, check_libcloud=False)
+        actual = _get_gcp_credentials(
+            module, require_valid_json=True, check_libcloud=False)
         expected = {'service_account_email': '',
                     'project_id': 'my-project',
                     'credentials_file': ''}