Repository: ansible/ansible-modules-extras

Issue:
bower_module thrown an error if bower_components dir already exists
I run next task in my playbook:

``` yaml
    - name: install bower vendor dependencies
      bower: path="/path/to/project" state=latest
```

And if `bower_components` dir not exists in `/path/to/project`, it download all dependencies and works fine, but if I run this task again, I get a next error:

```
< TASK: install bower vendor dependencies >
<transformation.server.ndv.net.ua> ESTABLISH CONNECTION FOR USER: srvadmin
<transformation.server.ndv.net.ua> REMOTE_MODULE bower path="/var/www/sites/transformation/web" state=latest
<transformation.server.ndv.net.ua> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/victor/.ansible/cp/ansible-ssh-%h-%p-%r" -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=srvadmin -o ConnectTimeout=10 transformation.server.ndv.net.ua /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340 && echo $HOME/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340'
<transformation.server.ndv.net.ua> PUT /tmp/tmpjZzhjm TO /home/srvadmin/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340/bower
<transformation.server.ndv.net.ua> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/victor/.ansible/cp/ansible-ssh-%h-%p-%r" -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=srvadmin -o ConnectTimeout=10 transformation.server.ndv.net.ua /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=vilcuzdrghmtlvfndvzibzqgxnyjsrmj] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-vilcuzdrghmtlvfndvzibzqgxnyjsrmj; LANG=C LC_CTYPE=C /usr/bin/python /home/srvadmin/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340/bower; rm -rf /home/srvadmin/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340/ >/dev/null 2>&1'"'"''

failed: [localhost] => {"failed": true, "parsed": false}

BECOME-SUCCESS-vilcuzdrghmtlvfndvzibzqgxnyjsrmj
Traceback (most recent call last):
  File "/home/srvadmin/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340/bower", line 1786, in <module>
    main()
  File "/home/srvadmin/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340/bower", line 173, in main
    installed, missing, outdated = bower.list()
  File "/home/srvadmin/.ansible/tmp/ansible-tmp-1431673951.26-168306551298340/bower", line 121, in list
    elif data['dependencies'][dep]['pkgMeta']['version'] != data['dependencies'][dep]['update']['latest']:
KeyError: 'version'
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
Shared connection to transformation.server.ndv.net.ua closed.
```

It looks like a bug.

P.S. I run playbook with `-vvvv` key. Remote machine is Ubuntu 12.04, bower is installed and work correctly if I run it manually with `bower install` or `bower update`

My workaround is to remove manually `bower_components` folder first if exists and then execute `install bower vendor dependencies` task.



Discussion:
@mwarkentin does this look good to you?

@mwarkentin sorry -- I thought this was a pull request.  Better question -- are you able to take a look at what could be happening to cause this?

@abadger, sorry, but I don't know Python very well. I use Ansible in my project for provision and deploy only.

No worries I wa sjust asking if the module author had time to look into this.

@bocharsky-bw Thanks for the report, sorry about the issue. I've been unable to reproduce using a simple bower.json locally. Could you test it out with a simple bower.json, and see if it's an issue? If not, is it possible for you to post your bower.json here or in a gist?

Are you using any private bower packages?

I'm wondering if you have a package you're trying to install which doesn't expose the `version` key in the metadata, which is expected here. Not sure if that's valid or not according to the bower spec.

It appears that [`version`](https://github.com/bower/bower.json-spec#version) is not actually something we should be relying on.

Here's a full list of what's available in `pkgMeta`: 

``` json
{
        "_direct": true,
        "_originalSource": "bootstrap",
        "_release": "3.3.4",
        "_resolution": {
            "commit": "a10eb60bc0b07b747fa0c4ebd8821eb7307bd07f",
            "tag": "v3.3.4",
            "type": "version"
        },
        "_source": "git://github.com/twbs/bootstrap.git",
        "_target": "~3.3.4",
        "dependencies": {
            "jquery": ">= 1.9.1"
        },
        "description": "The most popular front-end framework for developing responsive, mobile first projects on the web.",
        "homepage": "http://getbootstrap.com",
        "ignore": [
            "/.*",
            "_config.yml",
            "CNAME",
            "composer.json",
            "CONTRIBUTING.md",
            "docs",
            "js/tests",
            "test-infra"
        ],
        "keywords": [
            "css",
            "js",
            "less",
            "mobile-first",
            "responsive",
            "front-end",
            "framework",
            "web"
        ],
        "main": [
            "less/bootstrap.less",
            "dist/css/bootstrap.css",
            "dist/js/bootstrap.js",
            "dist/fonts/glyphicons-halflings-regular.eot",
            "dist/fonts/glyphicons-halflings-regular.svg",
            "dist/fonts/glyphicons-halflings-regular.ttf",
            "dist/fonts/glyphicons-halflings-regular.woff",
            "dist/fonts/glyphicons-halflings-regular.woff2"
        ],
        "name": "bootstrap",
        "version": "3.3.4"
    }
```

Perhaps using something like `_release` is safer? I think the other option is to always mark something as `outdated` if it's missing `version`, and always run update.

@mwarkentin Hi Michael!

My `bower.json` now looks like:

``` json
{
  "name": "transformation",
  "private": true,
  "dependencies": {
    "jquery": "~2.1",
    "bootstrap": "~3.3",
    "fontawesome": "~4.3",
    "ckeditor": "#full/4.4.6",
    "cropper": "~0.8.0",
    "jquery-number-mask": "~1.0.1"
  }
}
```

When I exec `bower install` or `bower update` manually - it works, but with ansible `bower_module` it fails. There is problem in my bower.json file?

@bocharsky-bw It looks like ckeditor doesn't expose `version` in their bower.json, which explains why this is blowing up for you: https://github.com/ckeditor/ckeditor-releases/blob/master/bower.json

I'll need to do some further investigation to figure out the right fix.

@mwarkentin OK, I'll be open for testing with my case if need.

@mwarkentin @bocharsky-bw I hit this same issue, but in a different way. Bower.json:

``` json
{
  "name": "atlas-colombia",
  "dependencies": {
    "ember": "1.12.0",
    "ember-cli-shims": "ember-cli/ember-cli-shims#0.0.3",
    "ember-cli-test-loader": "ember-cli-test-loader#0.1.3",
    "ember-data": "1.0.0-beta.18",
    "ember-load-initializers": "ember-cli/ember-load-initializers#0.1.4",
    "ember-qunit": "0.3.3",
    "ember-qunit-notifications": "0.0.7",
    "ember-resolver": "~0.1.15",
    "jquery": "^1.11.1",
    "loader.js": "ember-cli/loader.js#3.2.0",
    "qunit": "~1.17.1",
    "lodash": "~3.7.0",
    "pretender": "~0.6.0",
    "ember-inflector": "~1.3.1",
    "font-awesome": "~4.3.0",
    "cidcons": "https://github.com/cid-harvard/cidcons.git",
    "numeral": "~1.5.3",
    "nouislider": "~8.0.1"
  },
  "devDependencies": {
    "d3": "~3.5.5",
    "d3plus": "~1.7.3"
  },
  "resolutions": {
    "d3": "~3.5.5",
    "qunit": "~1.17.1"
  }
}
```

The failure happens at `"cidcons": "https://github.com/cid-harvard/cidcons.git",`, which might be because there are no releases on the github repo?

Backtrace: 

```
Traceback (most recent call last):
  File "/tmp/ansible-tmp-1436887794.47-254944865121850/bower", line 1786, in <module>
    main()
  File "/tmp/ansible-tmp-1436887794.47-254944865121850/bower", line 168, in main
    installed, missing, outdated = bower.list()
  File "/tmp/ansible-tmp-1436887794.47-254944865121850/bower", line 121, in list
    elif data['dependencies'][dep]['pkgMeta']['version'] != data['dependencies'][dep]['update']['latest']:
KeyError: 'version'
```

Where `dep` is `cidcons` and `data['dependencies'][dep]['pkgMeta']` looks like:

``` python
{u'name': u'cidcons', u'_source': u'https://github.com/cid-harvard/cidcons.git', u'_release': u'ff7bc5741e', u'_target': u'*', u'ignore': [], u'_originalSource': u'https://github.com/cid-harvard/cidcons.git', u'dependencies': {}, u'_resolution': {u'commit': u'ff7bc5741eb337d7ee6b6c8a1dfaca9d7384cfc3', u'type': u'branch', u'branch': u'master'}, u'authors': [u'Gus Wezerek <gwezerek@gmail.com>'], u'main': u'./css/cidcons.css', u'homepage': u'https://github.com/cid-harvard/cidcons'}
```

Maybe that if statement needs another branch?

The interesting thing is cidcons /does/ expose a "version": https://github.com/cid-harvard/cidcons/blob/master/bower.json

@makmanalp What about to use `"cidcons": "*",`? Does it work?

@bocharsky-bw I have to run out now but will test it out when I get back in an hour, thanks for the pointer!

@bocharsky-bw In my bower.json? How would it then know where the package is if I don't supply the url? I'm going to try to manually give it a version in the url.

More info, apparently bower.json doesn't need a version: http://bower.io/docs/creating-packages/#version

Specifying `"cidcons": "https://github.com/cid-harvard/cidcons.git#v0.3.0",` or `"cidcons": "https://github.com/cid-harvard/cidcons.git#0.3.0",`,  `"cidcons": "https://github.com/cid-harvard/cidcons.git",` or even `"cidcons": "cid-harvard/cidcons#0.3.0",` seems to make no difference. I found a quicker way to test this stuff, I just edit bower.json and look in the output of `bower list -j`. Somehow even when the cidcons repo includes a "version", the pkgMeta for it in bower never has version.

This makes me thing that the behavior for git URLs are different somehow, and that we probably shouldn't be checking for "version". We want "update" maybe?

OK, I removed "version" from the bower.json in cidcons, tagged a new release in github, specified it in bower.json like `"cidcons": "cid-harvard/cidcons#0.3.1",` and bower installed. This now caused the `bower list -j` output to have a "version" for cidcons. It's very odd that the package previously was not missing but didn't have a version either. I have solved my own issue but if you folks need help, I'd be happy to try to lend you a hand to reproduce more things.

It seems `cidcons` doesn't publish in Bower. Why you did not publish it?

Sorry for the spam, I think I'm one step closer to figuring it out. Now doing a deployment, it broke again. Putting a breakpoint at where it broke, I can print out the pkgMeta again:

```
{u'pkgMeta': {u'name': u'cidcons', u'_source': u'https://github.com/cid-harvard/cidcons.git', u'_release': u'ff7bc5741e', u'_target': u'*', u'ignore': [], u'_originalSource': u'https://github.com/cid-harvard/cidcons.git', u'dependencies': {}, u'_resolution': {u'commit': u'ff7bc5741eb337d7ee6b6c8a1dfaca9d7384cfc3', u'type': u'branch', u'branch': u'master'}, u'authors': [u'Gus Wezerek <gwezerek@gmail.com>'], u'main': u'./css/cidcons.css', u'homepage': u'https://github.com/cid-harvard/cidcons'}, u'endpoint': {u'source': u'cid-harvard/cidcons', u'name': u'cidcons', u'target': u'0.3.1'}, u'incompatible': True, u'versions': [u'0.3.1', u'0.3.0'], u'update': {u'target': u'0.3.1', u'latest': u'0.3.1'}, u'canonicalDir': u'/srv/frontend/bower_components/cidcons', u'dependencies': {}, u'nrDependants': 1}
```

I think I figured it out! Notice how incompatible is set to "true".

Missing is not set to true because a previous, older version is installed. Which causes it to skip over this line: https://github.com/ansible/ansible-modules-extras/blob/devel/packaging/language/bower.py#L119

But then this line fails because there is no "version": https://github.com/ansible/ansible-modules-extras/blob/devel/packaging/language/bower.py#L121

So it never gets to this line where the code checks for incompatible: https://github.com/ansible/ansible-modules-extras/blob/devel/packaging/language/bower.py#L123

I think the lesson here is that "version" (and "update", I think) are not always guaranteed to be in the pkgMeta, so the code should check for those before it does `https://github.com/ansible/ansible-modules-extras/blob/devel/packaging/language/bower.py#L121` on 121, and just skip to the next clause if so.

I can submit a pull request dealing with that unless there are any objections.

@bocharsky-bw Because I don't want / need the package to be in the registry - it's useless to most other people.

@makmanalp No objections - thanks for your help debugging. :smile: 


Resolving patch:
diff --git a/packaging/language/bower.py b/packaging/language/bower.py
--- a/packaging/language/bower.py
+++ b/packaging/language/bower.py
@@ -116,11 +116,15 @@ def list(self):
         data = json.loads(self._exec(cmd, True, False))
         if 'dependencies' in data:
             for dep in data['dependencies']:
-                if 'missing' in data['dependencies'][dep] and data['dependencies'][dep]['missing']:
+                dep_data = data['dependencies'][dep]
+                if dep_data.get('missing', False):
                     missing.append(dep)
-                elif data['dependencies'][dep]['pkgMeta']['version'] != data['dependencies'][dep]['update']['latest']:
+                elif \
+                  'version' in dep_data['pkgMeta'] and \
+                  'update' in dep_data and \
+                  dep_data['pkgMeta']['version'] != dep_data['update']['latest']:
                     outdated.append(dep)
-                elif 'incompatible' in data['dependencies'][dep] and data['dependencies'][dep]['incompatible']:
+                elif dep_data.get('incompatible', False):
                     outdated.append(dep)
                 else:
                     installed.append(dep)


Test patch: