Repository: ansible/ansible

Issue:
Jinja filters output trailing whitespace breaking idempotency
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
`lib/ansible/parsing/yaml/dumper.py:AnsibleDumper`

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel 6c101087ac) last updated 2017/03/29 16:09:54 (GMT +200)
  config file = 
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  python version = 2.7.9 (default, Jun 29 2016, 13:08:31) [GCC 4.9.2]

[and]
ansible 2.2.2.0 (stable-2.2 2273800f7c) last updated 2017/03/29 10:54:33 (GMT +200)
  lib/ansible/modules/core: (detached HEAD 31a1f19cd8) last updated 2017/03/29 16:07:02 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 921bc0d464) last updated 2017/03/29 14:42:48 (GMT +200)
```

##### CONFIGURATION
Ansible default. No changes.


##### OS / ENVIRONMENT
Isolated Debian Jessie VM on Qubes OS setup only for testing with Ansible devel.

##### SUMMARY
The `to_nice_json` filter and others like `indent` output trailing whitespace. That is not in itself a problem (although bad style). But in the case of `to_nice_json` it becomes a problem because it is potentially not idempotent which breaks CI which test for this property (e.g. DebOps).

Also note that the task itself outputs trailing whitespace (select the task output below).

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---

- hosts: localhost
  vars:
    input:
      - test: True
        test2:
          - 23
      - test: True

  tasks:

    - name: Jinja2 templating outputting trailing spaces which change depending
      debug:
        msg: "{{ (input | to_nice_json).split('\n') }}"

    # Workaround is part of https://github.com/debops/debops-playbooks/blob/master/templates/debops__tpl_macros.j2
    - name: Clean Jinja2 templating using workaround
      debug:
        msg: "{{ (input | to_nice_json | regex_replace(\"[ \\t\\r\\f\\v]+(\\n|$)\", \"\\1\")).split('\n') }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

```
TASK [Clean Jinja2 templating using workaround]
ok: [localhost] => {
    "changed": false, 
    "msg": [
        "[", 
        "    {", 
        "        \"test\": true,", 
        "        \"test2\": [", 
        "            23", 
        "        ]", 
        "    },", 
        "    {", 
        "        \"test\": true", 
        "    }", 
        "]"
    ]
}
```

Example role in CI: https://travis-ci.org/debops/ansible-apt/builds/216374820

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [Jinja2 template outputting trailing spaces which change depending on next element]
ok: [localhost] => {
    "changed": false, 
    "msg": [
        "[", 
        "    {", 
        "        \"test\": true, ", 
        "        \"test2\": [", 
        "            23", 
        "        ]", 
        "    }, ", 
        "    {", 
        "        \"test\": true", 
        "    }", 
        "]"
    ]
}
```

Example role in CI: https://travis-ci.org/debops/ansible-apt/builds/216355310#L824-L825

Jinja filters output trailing whitespace breaking idempotency
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
`lib/ansible/parsing/yaml/dumper.py:AnsibleDumper`

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel 6c101087ac) last updated 2017/03/29 16:09:54 (GMT +200)
  config file = 
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  python version = 2.7.9 (default, Jun 29 2016, 13:08:31) [GCC 4.9.2]

[and]
ansible 2.2.2.0 (stable-2.2 2273800f7c) last updated 2017/03/29 10:54:33 (GMT +200)
  lib/ansible/modules/core: (detached HEAD 31a1f19cd8) last updated 2017/03/29 16:07:02 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 921bc0d464) last updated 2017/03/29 14:42:48 (GMT +200)
```

##### CONFIGURATION
Ansible default. No changes.


##### OS / ENVIRONMENT
Isolated Debian Jessie VM on Qubes OS setup only for testing with Ansible devel.

##### SUMMARY
The `to_nice_json` filter and others like `indent` output trailing whitespace. That is not in itself a problem (although bad style). But in the case of `to_nice_json` it becomes a problem because it is potentially not idempotent which breaks CI which test for this property (e.g. DebOps).

Also note that the task itself outputs trailing whitespace (select the task output below).

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---

- hosts: localhost
  vars:
    input:
      - test: True
        test2:
          - 23
      - test: True

  tasks:

    - name: Jinja2 templating outputting trailing spaces which change depending
      debug:
        msg: "{{ (input | to_nice_json).split('\n') }}"

    # Workaround is part of https://github.com/debops/debops-playbooks/blob/master/templates/debops__tpl_macros.j2
    - name: Clean Jinja2 templating using workaround
      debug:
        msg: "{{ (input | to_nice_json | regex_replace(\"[ \\t\\r\\f\\v]+(\\n|$)\", \"\\1\")).split('\n') }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

```
TASK [Clean Jinja2 templating using workaround]
ok: [localhost] => {
    "changed": false, 
    "msg": [
        "[", 
        "    {", 
        "        \"test\": true,", 
        "        \"test2\": [", 
        "            23", 
        "        ]", 
        "    },", 
        "    {", 
        "        \"test\": true", 
        "    }", 
        "]"
    ]
}
```

Example role in CI: https://travis-ci.org/debops/ansible-apt/builds/216374820

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [Jinja2 template outputting trailing spaces which change depending on next element]
ok: [localhost] => {
    "changed": false, 
    "msg": [
        "[", 
        "    {", 
        "        \"test\": true, ", 
        "        \"test2\": [", 
        "            23", 
        "        ]", 
        "    }, ", 
        "    {", 
        "        \"test\": true", 
        "    }", 
        "]"
    ]
}
```

Example role in CI: https://travis-ci.org/debops/ansible-apt/builds/216355310#L824-L825



Discussion:
Files identified in the description:
* [lib/ansible/parsing/yaml/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/__init__.py)
* [lib/ansible/parsing/yaml/constructor.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/constructor.py)
* [lib/ansible/parsing/yaml/dumper.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/dumper.py)
* [lib/ansible/parsing/yaml/loader.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/loader.py)
* [lib/ansible/parsing/yaml/objects.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/objects.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [lib/ansible/parsing/utils/yaml.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/utils/yaml.py)
* [lib/ansible/parsing/yaml/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/__init__.py)
* [lib/ansible/parsing/yaml/constructor.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/constructor.py)
* [lib/ansible/parsing/yaml/dumper.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/dumper.py)
* [lib/ansible/parsing/yaml/loader.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/loader.py)
* [lib/ansible/parsing/yaml/objects.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/objects.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
This is not an implementation detail of ansible, but of the way that `json.dumps` from the python std library operates when using `indent`.

As such, this would be considered an upstream python bug, and not a bug in ansible.

If you have further questions please stop by IRC or the mailing list:

   * IRC: #ansible on irc.freenode.net
   * mailing list: https://groups.google.com/forum/#!forum/ansible-project 
Files identified in the description:
* [lib/ansible/parsing/yaml/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/__init__.py)
* [lib/ansible/parsing/yaml/constructor.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/constructor.py)
* [lib/ansible/parsing/yaml/dumper.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/dumper.py)
* [lib/ansible/parsing/yaml/loader.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/loader.py)
* [lib/ansible/parsing/yaml/objects.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/objects.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [lib/ansible/parsing/utils/yaml.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/utils/yaml.py)
* [lib/ansible/parsing/yaml/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/__init__.py)
* [lib/ansible/parsing/yaml/constructor.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/constructor.py)
* [lib/ansible/parsing/yaml/dumper.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/dumper.py)
* [lib/ansible/parsing/yaml/loader.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/loader.py)
* [lib/ansible/parsing/yaml/objects.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/parsing/yaml/objects.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
This is not an implementation detail of ansible, but of the way that `json.dumps` from the python std library operates when using `indent`.

As such, this would be considered an upstream python bug, and not a bug in ansible.

If you have further questions please stop by IRC or the mailing list:

   * IRC: #ansible on irc.freenode.net
   * mailing list: https://groups.google.com/forum/#!forum/ansible-project 

Resolving patch:
diff --git a/lib/ansible/plugins/filter/core.py b/lib/ansible/plugins/filter/core.py
--- a/lib/ansible/plugins/filter/core.py
+++ b/lib/ansible/plugins/filter/core.py
@@ -87,7 +87,7 @@ def to_json(a, *args, **kw):
 def to_nice_json(a, indent=4, *args, **kw):
     '''Make verbose, human readable JSON'''
     try:
-        return json.dumps(a, indent=indent, sort_keys=True, cls=AnsibleJSONEncoder, *args, **kw)
+        return json.dumps(a, indent=indent, sort_keys=True, separators=(',', ': '), cls=AnsibleJSONEncoder, *args, **kw)
     except Exception as e:
         # Fallback to the to_json filter
         display.warning(u'Unable to convert data using to_nice_json, falling back to to_json: %s' % to_text(e))


Test patch: