Repository: ansible/ansible

Issue:
Lookup plugins cannot return list to Jinja2 template
##### Issue Type:

Bug
##### Ansible Version:

ansible 1.8.2
(Also tested by @bcoca on current devel)
##### Environment:

N/A
##### Summary:

If a lookup plugin returns a list `['one', 'two', 'three']` this is stringified and returned to a Jinja2 template as a string `one,two,three`.
##### Steps To Reproduce:
###### environment

```
export ANSIBLE_LOOKUP_PLUGINS=`pwd`
```
###### lookup mini.py

``` python
from ansible import utils, errors
import os

class LookupModule(object):

    def __init__(self, basedir=None, **kwargs):
        self.basedir = basedir

    def run(self, terms, inject=None, **kwargs):

        ret = ['gato', 'chat', 'katze', 'miau' ]

        return ret
```
###### playbook

``` yaml
- hosts:
   - localhost
  gather_facts: False
  connection: local
  tasks:
  - action: template src=t.in dest=/tmp/t.out
```
###### template

```
#-- result from lookup()
{% for k in lookup('mini', 'blah')  %}
        {{ k }}
{% endfor %}

#-- result from lookup() | list
{% for k in lookup('mini', 'blah') | list  %}
        {{ k }}
{% endfor %}

#-- lookup().split()
{{ lookup('mini', 'blah').split(',')  }}

#-- Jinja2 list
{% for k in  ['uno', 'dos', 'tres' ] %}
        {{ k }}
{% endfor %}
```
##### Expected Results:

I expected the following from `lookup('mini', 'blah')`

```
    gato
    chat
    katze
    miau
```
##### Actual Results:

Results from running above playbook:

```
#-- result from lookup()
        g
        a
        t
        o
        ,
        c
        h
        a
        t
        ,
        k
        a
        t
        z
        e
        ,
        m
        i
        a
        u

#-- result from lookup() | list
        g
        a
        t
        o
        ,
        c
        h
        a
        t
        ,
        k
        a
        t
        z
        e
        ,
        m
        i
        a
        u

#-- lookup().split()
['gato', 'chat', 'katze', 'miau']

#-- Jinja2 list
        uno
        dos
        tres
```



Discussion:
Returning a comma separated list is explicitly coded behaviour for templated {{ lookup() }}'s
See https://github.com/ansible/ansible/blob/devel/lib/ansible/utils/template.py#L102

This came up before, and I never understood that behaviour, but it was what it was.
(In the mean time I rely on that btw. perhaps a listify() filter can help?)

verified the issue, |list filter does not seem to help

Commenting out that line indeed results in lookup() doing what I would expect it to do.

What would listify() do? I assume a split()? That's not good enough as it would mean I'd have to settle on something I can split on.. Also, Jinja2 has .split(), which works, but as I said: not good enough, I think.

.split(',') seems to be the way ansible expects to deal with lookup output

`.split(',')` is useless if the strings in the "list" contain comma; this is true for whatever character we settle on, right? :)

Changing this means that everyone already using .split would then run into issues.

I've seen .split used in this situation quite a bit.

True, but the .split is probably being used because this was wrong to start with...

I'm not asking for this to be changed. I am, however, asking for an elegant solution even if that's classified as a workaround. :)

Would something like this be a compromise people can live with? `lookup()` functionality remains as is. `llookup()` (**l**ist lookup) returns as a list. (I'm quite sure one of you gurus can improve on this code.)

``` patch
diff --git a/lib/ansible/utils/template.py b/lib/ansible/utils/template.py
index 9556b8f..0ce21e6 100644
--- a/lib/ansible/utils/template.py
+++ b/lib/ansible/utils/template.py
@@ -84,7 +84,7 @@ _LISTRE = re.compile(r"(\w+)\[(\d+)\]")
 JINJA2_OVERRIDE = '#jinja2:'
 JINJA2_ALLOWED_OVERRIDES = ['trim_blocks', 'lstrip_blocks', 'newline_sequence', 'keep_trailing_newline']

-def lookup(name, *args, **kwargs):
+def llookup(name, *args, **kwargs):
     from ansible import utils
     instance = utils.plugins.lookup_loader.get(name.lower(), basedir=kwargs.get('basedir',None))
     tvars = kwargs.get('vars', None)
@@ -98,12 +98,16 @@ def lookup(name, *args, **kwargs):
             raise errors.AnsibleUndefinedVariable("One or more undefined variables: %s" % str(e))
         except Exception, e:
             raise errors.AnsibleError('Unexpected error in during lookup: %s' % e)
-        if ran:
-            ran = ",".join(ran)
         return ran
     else:
         raise errors.AnsibleError("lookup plugin (%s) not found" % name)

+def lookup(name, *args, **kwargs):
+    ran = llookup(name, *args, **kwargs)
+    if ran:
+        ran = ",".join(ran)
+    return ran
+
 def template(basedir, varname, templatevars, lookup_fatal=True, depth=0, expand_lists=True, convert_bare=False, fail_on_undefined=False, filter_fatal=True):
     ''' templates a data structure by traversing it and substituting for other data structures '''
     from ansible import utils
@@ -215,6 +219,9 @@ def template_from_file(basedir, path, vars, vault_password=None):
     realpath = utils.path_dwim(basedir, path)
     loader=jinja2.FileSystemLoader([basedir,os.path.dirname(realpath)])

+    def my_llookup(*args, **kwargs):
+        kwargs['vars'] = vars
+        return llookup(*args, basedir=basedir, **kwargs)
     def my_lookup(*args, **kwargs):
         kwargs['vars'] = vars
         return lookup(*args, basedir=basedir, **kwargs)
@@ -224,6 +231,7 @@ def template_from_file(basedir, path, vars, vault_password=None):
     environment = jinja2.Environment(loader=loader, trim_blocks=True, extensions=_get_extensions())
     environment.filters.update(_get_filters())
     environment.globals['lookup'] = my_lookup
+    environment.globals['llookup'] = my_llookup
     environment.globals['finalize'] = my_finalize
     if fail_on_undefined:
         environment.undefined = StrictUndefined
```

Running the test I created above with this new function produces what I think is correct:

```
#-- result from llookup()
        gato
        chat
        katze
        miau

#-- result from llookup() | list
        gato
        chat
        katze
        miau
```


Resolving patch:
diff --git a/lib/ansible/utils/template.py b/lib/ansible/utils/template.py
--- a/lib/ansible/utils/template.py
+++ b/lib/ansible/utils/template.py
@@ -89,6 +89,8 @@ def lookup(name, *args, **kwargs):
     instance = utils.plugins.lookup_loader.get(name.lower(), basedir=kwargs.get('basedir',None))
     tvars = kwargs.get('vars', None)
 
+    wantlist = kwargs.pop('wantlist', False)
+
     if instance is not None:
         try:
             ran = instance.run(*args, inject=tvars, **kwargs)
@@ -98,7 +100,7 @@ def lookup(name, *args, **kwargs):
             raise errors.AnsibleUndefinedVariable("One or more undefined variables: %s" % str(e))
         except Exception, e:
             raise errors.AnsibleError('Unexpected error in during lookup: %s' % e)
-        if ran:
+        if ran and not wantlist:
             ran = ",".join(ran)
         return ran
     else:


Test patch: