Repository: openshift/openshift-ansible

Issue:
Deploy PersistentVolume definitions FAILS
```
TASK [roles/openshift_persistent_volumes : Deploy PersistentVolume definitions] 
fatal: [10.57.50.187]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleFilterError: Failed to convert: %s"}
```

Here is the call to PersistentVolume role:

```
- name: Create persistent volumes/claims
  hosts: masters[0]
  vars:
    openshift_deployment_type: origin
    persistent_volumes:
    - name: "persistent-log-storage"
      capacity: "10Gi"
      access_modes:
      - "ReadWriteMany"
      storage:
        nfs:
          server: "www.server.com"
          path: "/drm-persistent-storage"
    persistent_volume_claims:
    - name: "persistent-log-storage-claim"
      capacity: "5Gi"
      access_modes:
      - "ReadWriteMany"
  roles:
  - role: roles/openshift_persistent_volumes`
```

TAG: openshift-ansible-3.3.16-1
Destination machine: CentOS Linux release 7.2.1511 (Core)
3.10.0-327.28.2.el7.x86_64 #1 SMP



Discussion:
@deromka I can't tell what's wrong from the output you provided (though I do appreciate it!)

> FAILED! => {"changed": false, "failed": true, "msg": "AnsibleFilterError: Failed to convert: %s"}

You did just help me uncover an error in our filter plugin. See that `%s` in there? That **should** be the error message that was raised when the Filter Error happened, but we have a typo in our filter module

We're doing this:

> raise errors.AnsibleFilterError('Failed to convert: %s', my_e)

However, according to the Ansible source code:

> def **init**(self, message="", obj=None, show_content=True, suppress_extended_error=False): 

the second parameter to the `AnsibleFilterError` exception should be an `obj` which is _"some subclass of ansible.parsing.yaml.objects.AnsibleBaseYAMLObject, which should be returned by the DataLoader() class"_.

What we should be doing is this:

> raise errors.AnsibleFilterError('Failed to convert: %s' % my_e)

So that the string substitution actually happens. I'm going to file a PR for this error and then ask you to test again.

In the mean time, can you try applying this patch manually and re-running your playbook?

https://gist.github.com/tbielawa/73a41214cb02954f54bfc0858fa2ecb8

You can apply it in one command like this:

> curl https://gist.githubusercontent.com/tbielawa/73a41214cb02954f54bfc0858fa2ecb8/raw/e5a92e56198f44b58f6b051fa9c45023c46f228f/0001-Fix-string-substitution-error-in-the-to_padded_yaml-.patch | git apply -

Removing labels until we can pinpoint why the filter is failing.

I have applied patch, here is the real error:

```
TASK [roles/openshift_persistent_volumes : Deploy PersistentVolume definitions] 
fatal: [10.57.50.187]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleFilterError: Failed to convert: cannot represent an object: ReadWriteMany"}

```

Someone has an idea regarding the cause?

@deromka Is this a custom playbook that is leveraging the role?

Yes, here is the call to role

``` yaml
- name: Create persistent volumes/claims
  hosts: masters[0]
  vars:
    openshift_deployment_type: origin
    persistent_volumes:
    - name: "persistent-log-storage"
      capacity: "10Gi"
      access_modes:
      - "ReadWriteMany"
      storage:
        nfs:
          server: "www.server.com"
          path: "/drm-persistent-storage"
    persistent_volume_claims:
    - name: "persistent-log-storage-claim"
      capacity: "5Gi"
      access_modes:
      - "ReadWriteMany"
  roles:
  - role: roles/openshift_persistent_volumes
```

**edited by tbielawa to properly display the indented yaml snippet**

@tbielawa @abutcher I suspect it is a case of a missing dependency

@deromka are you running ansible through Python 3.x?

@abutcher the `openshift_persistent_volumes` role has your name on it, do you have any ideas what might be going on here? @deromka's snippet matches up with the [example given](https://github.com/openshift/openshift-ansible/tree/master/roles/openshift_persistent_volumes#example-playbook) in `roles/openshift_persistent_volumes/README.md` almost 1-for-1. 

@tbielawa I'm using python 2.7.x

On Tue, Sep 6, 2016, 20:33 Tim Bielawa notifications@github.com wrote:

> @abutcher https://github.com/abutcher the openshift_persistent_volumes
> role has your name on it, do you have any ideas what might be going on
> here? @deromka https://github.com/deromka's snippet matches up with the example
> given
> https://github.com/openshift/openshift-ansible/tree/master/roles/openshift_persistent_volumes#example-playbook
> in roles/openshift_persistent_volumes/README.md almost 1-for-1.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/openshift/openshift-ansible/issues/2401#issuecomment-245027336,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AAsgCf6TbcaguflZPUdinNpxCL3SPKZpks5qnaP8gaJpZM4Jzslh
> .

For the record, I tried to reproduce this error following this procedure:
1. Copy @deromka's example to a yaml file (`gh2401.yaml`)
2. `touch` the file `filter_plugins/__init__.yaml`
3. Open ipython in the root dir of the openshift-ansible repo:

``` python
>>> import filter_plugins.oo_filters
>>> import yaml
>>> f = filter_plugins.oo_filters.FilterModule()
>>> with open('gh2401.yaml', 'r') as y:
    rolecall = yaml.safe_load(y)
>>> print f.to_padded_yaml(rolecall)
- hosts: masters[0]
  name: Create persistent volumes/claims
  roles:
  - role: roles/openshift_persistent_volumes
  vars:
    openshift_deployment_type: origin
    persistent_volume_claims:
    - access_modes:
      - ReadWriteMany
      capacity: 5Gi
      name: persistent-log-storage-claim
    persistent_volumes:
    - access_modes:
      - ReadWriteMany
      capacity: 10Gi
      name: persistent-log-storage
      storage:
        nfs:
          path: /drm-persistent-storage
          server: www.server.com
```

I did some deep-debugging and hacked the system `yaml` library. I switched the `yaml.representer` module's `represent_undefined` method to include the class of the un-representable object like so:

``` python
# /usr/lib64/python2.7/site-packages/yaml/representer.py
def represent_undefined(self, data):
-    raise RepresenterError("cannot represent an object: %s" % data)
+    raise RepresenterError("cannot represent an object: %s (type: %s)" % (data, type(data)))
```

The result when I run the playbook again shows exactly what's happening here:

```
TASK [openshift_persistent_volumes : Deploy PersistentVolume definitions] ******
Tuesday 06 September 2016  14:55:07 -0700 (0:00:00.339)       0:00:27.961 *****
fatal: [m01.example.com]: FAILED! => {
    "changed": false, 
    "failed": true
}

MSG:

AnsibleFilterError: Failed to convert: cannot represent an object: ReadWriteMany (type: <class 'ansible.parsing.yaml.objects.AnsibleUnicode'>)
```

When ansible is parsing the playbook the tokens are getting transformed into a special Ansible unicode object, `ansible.parsing.yaml.objects.AnsibleUnicode`. This explains why the standard YAML parser raises the exception `cannot represent an object: ....`, because it doesn't know how to serialize `AnsibleUnicode` objects.

Per the Ansible source code, an Ansible `AnsibleUnicode` object inherits from two parent classes:
- `AnsibleBaseYAMLObject`
- `text_type` (which comes from `ansible.compat.six`

`text_type` comes from the Python **six** (2*3 = 6) utility for writing portable Python 2/3 code. For Python 2.x `text_type` is just an alias for `unicode`.

And `AnsibleBaseYAMLObject`s have the following docstring

> the base class used to sub-class python built-in objects so that we can add attributes to them during yaml parsing

Through some experimentation I've discovered that `AnsibleUnicode` objects can be created just like strings:

``` python
>>> s = ansible.parsing.yaml.objects.AnsibleUnicode("foo")
>>> type(s)
ansible.parsing.yaml.objects.AnsibleUnicode

>>> s1 = str(s)
>>> type(s1)
str
```

So it seemed natural to me that in `filter_plugins/oo_filters.py` we could just include a little `isinstance(data, AnsibleUnicode)` condition that changes `data` to a normal `str` object

``` python
def to_padded_yaml(data, level=0, indent=2, **kw):
    # ...
    try:
        if isinstance(data, AnsibleUnicode):
            data = str(data)
        transformed = yaml.safe_dump(data, indent=indent, allow_unicode=True, default_flow_style=False, **kw)
        # ...
```

But that had no effect. I followed that up with a blanket transformation to string:

``` python
def to_padded_yaml(data, level=0, indent=2, **kw):
    # ...
    try:
        data = str(data)
        transformed = yaml.safe_dump(data, indent=indent, allow_unicode=True, default_flow_style=False, **kw)
        # ...
```

That's when things got interesting. The original task which was failing in this issue now succeeds!

```
TASK [openshift_persistent_volumes : Deploy PersistentVolume definitions] ******
Tuesday 06 September 2016  15:23:37 -0700 (0:00:00.338)       0:00:27.802 *****
ok: [m01.example.com]
```

However, the `Create PersistentVolumes` task now fails with a YAML parsing error:

```
TASK [openshift_persistent_volumes : Create PersistentVolumes] *****************
Tuesday 06 September 2016  15:21:43 -0700 (0:00:00.553)       0:00:40.324 *****
fatal: [m01.example.com]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "create", 
        "-f", 
        "/tmp/openshift-ansible-9m1JaBc/persistent-volumes.yml", 
        "--config=/tmp/openshift-ansible-9m1JaBc/admin.kubeconfig"
    ], 
    "delta": "0:00:00.345450", 
    "end": "2016-09-06 18:21:43.643238", 
    "failed": true, 
    "failed_when_result": true, 
    "rc": 1, 
    "start": "2016-09-06 18:21:43.297788", 
    "warnings": []
}

STDERR:

yaml: line 13: could not find expected ':'
```

I enabled Ansible's `ANSIBLE_KEEP_REMOTE_FILES` environment variable and re-ran the playbook. I found that the templated file looked as such:

``` yaml
[root@m01 tmp]# cat /tmp/openshift-ansible-h6XD1rh/persistent-volumes.yml
---
apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: "registry-volume"
  spec:
    capacity:
      storage: "5Gi"
    accessModes: 
    '[u''ReadWriteMany'']'
    nfs: 
      '{u''path'': u''/var/exports/registry'', u''server'': u''nfs.example.com''}'
```

Looks like we're somewhat closer to a working solution.

Followup idea from the previous brain-dump

Perhaps we could utilize the YAML libraries `representers` system somehow here:
- http://stackoverflow.com/a/26794274/6490583
- http://pyyaml.org/wiki/PyYAMLDocumentation#Constructorsrepresentersresolvers

**edit** Also, our Jinja template seems to expect `accessModes` to be single strings, but we're accepting them as arrays. Hence why the templated YAML file shows `ReadWriteMany` as `'[u''ReadWriteMany'']'`.

Hold up. I'm going to try making that loop instead.

And the `nfs` key is obviously getting messed up by the blanket `str` application. That needs fixing.

Made a breakthrough, the values (or keys and values in the case of dicts) are being encoded as the special Ansible objects. So a simple `str(obj)` or `dict(obj)` is insufficient. Iterating over list contents/dict key-values and casting them each to `unicode` has produced better results. Time to cleanup this test work and see if I can make something intelligent out of it.

@deromka can you please try applying this patch? I think it might fix the issue you're running into:

https://gist.github.com/tbielawa/535f3f7d26e00226f768616de5f60eb6

> curl https://gist.githubusercontent.com/tbielawa/535f3f7d26e00226f768616de5f60eb6/raw/4207f3785a9e0b5eaa47c791f1238455439326c2/0001-Fix-for-objects-which-can-t-be-converted-by-the-YAML.patch | git apply -

In summary:
- Ansible is taking your playbook vars and converting the tokens into special Ansible objects
- openshift-ansible is trying to write out a YAML file to feed to `oc create` to create  your Persistent Volumes
- This patch takes those special Ansible objects (which can not be serialized by the native YAML parser) and turning them back into normal strings

@tbielawa I confirm, the patch solves this issue

@deromka thanks for verifying the patch. We're working on a little refactor of it now so it covers some edge cases I missed before. Thanks for your patience while we sort this out.

Sure, Thank you @tbielawa 

@tbielawa assuming that each list or dict consist of strings is probably a bit presumptive. I would expect there to be a new helper within Ansible to properly handle serializing AnsibleString to yaml properly. Tricky part will be making sure that it works for multiple ansible versions.

@tbielawa I did a quick grep on the ansible repo, looks like there is an AnsibleDumper that is referenced within lib/ansible/plugins/filter/core.py


Resolving patch:
diff --git a/filter_plugins/oo_filters.py b/filter_plugins/oo_filters.py
--- a/filter_plugins/oo_filters.py
+++ b/filter_plugins/oo_filters.py
@@ -16,6 +16,7 @@
 import re
 import json
 import yaml
+from ansible.parsing.yaml.dumper import AnsibleDumper
 from ansible.utils.unicode import to_unicode
 from urlparse import urlparse
 
@@ -621,7 +622,9 @@ def to_padded_yaml(data, level=0, indent=2, **kw):
             return ""
 
         try:
-            transformed = yaml.safe_dump(data, indent=indent, allow_unicode=True, default_flow_style=False, **kw)
+            transformed = yaml.dump(data, indent=indent, allow_unicode=True,
+                                    default_flow_style=False,
+                                    Dumper=AnsibleDumper, **kw)
             padded = "\n".join([" " * level * indent + line for line in transformed.splitlines()])
             return to_unicode("\n{0}".format(padded))
         except Exception as my_e:


Test patch: