Repository: ansible/ansible

Issue:
List order is lost using with_items
##### ISSUE TYPE
Bug Report

##### COMPONENT NAME
core

##### ANSIBLE VERSION
```
ansible 2.2.1.0
ansible 2.3.0
```

##### CONFIGURATION


##### OS / ENVIRONMENT
CentOS Linux 7

##### SUMMARY
List order is lost using with_items

##### STEPS TO REPRODUCE
```yaml
---
- hosts: localhost
  vars:
    with_dict_test:
      - { key: 'one', value: 1 }
      - { key: 'two', value: 2 }
      - { key: 'three', value: 3 }
      - { key: 'four', value: 4 }
      - { key: 'five', value: 5 }
      - { key: 'six', value: 6 }
  tasks:
      - name: with_dict test
        debug: msg="{{item.key}} --> {{item.value}}"
        with_items: "{{ with_dict_test}}"
```

##### EXPECTED RESULTS
```
ok: [localhost] => (item={u'value': 1, u'key': u'one'}) => {
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'value': 2, u'key': u'two'}) => {
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'value': 3, u'key': u'three'}) => {
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'value': 4, u'key': u'four'}) => {
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'value': 5, u'key': u'five'}) => {
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'value': 6, u'key': u'six'}) => {
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}
```

##### ACTUAL RESULTS
```
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}

```




Discussion:
I cannot reproduce, i ran 10 times and get the same order every time

```
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "item": {
        "key": "three",
        "value": 3
    },
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "item": {
        "key": "four",
        "value": 4
    },
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "item": {
        "key": "five",
        "value": 5
    },
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "item": {
        "key": "six",
        "value": 6
    },
    "msg": "six --> 6"
}

```

That is with_items, if using with_dict, dictionaries are unordered in python so I would expect the order to change.
Did it worked well for you under CentOS7? For me this test playbook works good on RHEL6 but return items in random order everytime on all 3 CentOS7 servers i tested (using the same ansible version 2.2.1.0). So it's maybe dependencies or something... What else can I check?

I used centos 5 through 7, no changes. I don't know of anything that would produce this unless you are modifying python core itself.
We are able to reproduce this issue after upgrading to 2.2. If machine is loaded it happens more often. So i recommend run something that loads your CPU in the background

I attach a gist with Alpine Dockerfile to be able to reproduce faster. It also happens on Ubuntu 16.04

To run the test
```bash
git clone https://gist.github.com/136228d60ea205bd1977c569f37b7584.git
cd 136228d60ea205bd1977c569f37b7584
docker build . -t ansible-issue
docker run ansible-issue
```
This will run a play in endless while loop and hashes the stdout so we can define easier the changed out.
Look at the [gist](https://gist.github.com/ahelal/136228d60ea205bd1977c569f37b7584) for more details
```
0c4a249410c6afc13ff04a4c08f46c3a  -
0c4a249410c6afc13ff04a4c08f46c3a  -
d7c08ff2ff85d2ff21b504d585e3d2bc  -
0c4a249410c6afc13ff04a4c08f46c3a  -
0c4a249410c6afc13ff04a4c08f46c3a  -
```

Hope that helps
I can confirm the same behavior on Arch Linux with ansible 2.2.1.0. Everything works perfectly fine until you introduce some background load (running two instances of `cat /dev/urandom | gzip > /dev/null` makes it happen for me)
I confirm problem with 2.2.1.0, 2.2.0.0, python 2.7.12. Latest non failing one is 2.1.4.0. CPU load definitely is a concern.
```
---
- hosts: localhost
  connection: local
  gather_facts: False

  tasks:
      - debug: msg="{{ item }}"
        with_items:
          - "One"
          - "Two"
          - "Three"
          - "Four"
          - "Five"
```
Simple way to confirm the problem is to run this play in an endless cycle.
So I still cannot reproduce, I have tried loading my box but no dice. 
```
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
....
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
ed5c688e29fd6e298cd1a6aa5d79bf4e  -
```
^ while running 3 of the `cat`,  encoding video and running 2 games at the same time.

I have looked at the code and identified a possible race condition in the queuing code but I'm not sure that is the problem (reporting the output not queuing the work).

This patch might fix the issue:

```
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
index 99d611417..cbe2ea3ad 100644
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -291,7 +291,7 @@ class TaskExecutor:
                 templar = Templar(loader=self._loader, shared_loader_obj=self._shared_loader_obj, variables=self._job_vars)
                 res['_ansible_item_label'] = templar.template(label)

-            self._rslt_q.put(TaskResult(self._host.name, self._task._uuid, res), block=False)
+            self._rslt_q.put(TaskResult(self._host.name, self._task._uuid, res), block=True)
             results.append(res)
             del task_vars[loop_var]

```
Just to verify some assumptions:

   When the output is not ordered correctly, the wrong output is usually different or always the same?
   (I assume usually different/random, but just to be sure.)

Are the folks that are seeing failures, are the tests running bare metal or vm? (and/or a container)
  
Which python versions?
I can reproduce this now. On my system, running pigz (parallel gzip) seems to make it easier to trigger 

```
cat /dev/urandom | pigz -- > /dev/null
```

Using the playbook test.yml

``` yaml
---
- hosts: localhost
  connection: local
  vars:
    with_dict_test:
      - { key: 'one', value: 1 }
      - { key: 'two', value: 2 }
      - { key: 'three', value: 3 }
      - { key: 'four', value: 4 }
      - { key: 'five', value: 5 }
      - { key: 'six', value: 6 }
  tasks:
      - name: with_dict test
        debug: msg="{{item.key}} --> {{item.value}}"
        with_items: "{{ with_dict_test}}"
        register: results

      - name: debug results
        debug:
          var: results
```

and the test script entry_debug.sh:

``` shell
#!/bin/sh

AP=ansible-playbook
while /bin/true; do
    $AP -i localhost, test.yml --connection=local |tee -a out.log | md5sum
    #ansible-playbook -i localhost, play.yml --connection=local;
done
```

I start seeing the stdout of the 'ansible-playbook' invocations showing the with_items debug statements out of order. The registered 'results' var however, does show the result in order.

snippet from out.log:

```

PLAY [localhost] *********************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [with_dict test] ****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}

TASK [debug results] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "changed": false, 
    "results": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "one", 
                    "value": 1
                }, 
                "msg": "one --> 1"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "two", 
                    "value": 2
                }, 
                "msg": "two --> 2"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "three", 
                    "value": 3
                }, 
                "msg": "three --> 3"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "four", 
                    "value": 4
                }, 
                "msg": "four --> 4"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "five", 
                    "value": 5
                }, 
                "msg": "five --> 5"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "six", 
                    "value": 6
                }, 
                "msg": "six --> 6"
            }
        ]
    }
}

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   
```






Testing with devel branch + patch mentioned in https://github.com/ansible/ansible/issues/21008#issuecomment-281139714

Still seems to break. Afaict the behavior is about the same.


output:

```

PLAY [localhost] *********************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [with_dict test] ****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}

TASK [debug results] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "changed": false, 
    "results": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "one", 
                    "value": 1
                }, 
                "msg": "one --> 1"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "two", 
                    "value": 2
                }, 
                "msg": "two --> 2"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "three", 
                    "value": 3
                }, 
                "msg": "three --> 3"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "four", 
                    "value": 4
                }, 
                "msg": "four --> 4"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "five", 
                    "value": 5
                }, 
                "msg": "five --> 5"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "six", 
                    "value": 6
                }, 
                "msg": "six --> 6"
            }
        ]
    }
}

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   


PLAY [localhost] *********************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [with_dict test] ****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}

TASK [debug results] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "changed": false, 
    "results": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "one", 
                    "value": 1
                }, 
                "msg": "one --> 1"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "two", 
                    "value": 2
                }, 
                "msg": "two --> 2"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "three", 
                    "value": 3
                }, 
                "msg": "three --> 3"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "four", 
                    "value": 4
                }, 
                "msg": "four --> 4"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "five", 
                    "value": 5
                }, 
                "msg": "five --> 5"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "six", 
                    "value": 6
                }, 
                "msg": "six --> 6"
            }
        ]
    }
}

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   


PLAY [localhost] *********************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [with_dict test] ****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}

TASK [debug results] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "changed": false, 
    "results": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "one", 
                    "value": 1
                }, 
                "msg": "one --> 1"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "two", 
                    "value": 2
                }, 
                "msg": "two --> 2"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "three", 
                    "value": 3
                }, 
                "msg": "three --> 3"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "four", 
                    "value": 4
                }, 
                "msg": "four --> 4"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "five", 
                    "value": 5
                }, 
                "msg": "five --> 5"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "six", 
                    "value": 6
                }, 
                "msg": "six --> 6"
            }
        ]
    }
}

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   

```
testing against https://github.com/ansible/ansible/compare/stash_task_fields_for_task_result

output:

```

PLAY [localhost] *********************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [with_dict test] ****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}

TASK [debug results] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "changed": false, 
    "results": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "one", 
                    "value": 1
                }, 
                "msg": "one --> 1"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "two", 
                    "value": 2
                }, 
                "msg": "two --> 2"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "three", 
                    "value": 3
                }, 
                "msg": "three --> 3"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "four", 
                    "value": 4
                }, 
                "msg": "four --> 4"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "five", 
                    "value": 5
                }, 
                "msg": "five --> 5"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "six", 
                    "value": 6
                }, 
                "msg": "six --> 6"
            }
        ]
    }
}

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   


PLAY [localhost] *********************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [with_dict test] ****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'key': u'one', u'value': 1}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "one", 
        "value": 1
    }, 
    "msg": "one --> 1"
}
ok: [localhost] => (item={u'key': u'two', u'value': 2}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "two", 
        "value": 2
    }, 
    "msg": "two --> 2"
}
ok: [localhost] => (item={u'key': u'three', u'value': 3}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "three", 
        "value": 3
    }, 
    "msg": "three --> 3"
}
ok: [localhost] => (item={u'key': u'four', u'value': 4}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "four", 
        "value": 4
    }, 
    "msg": "four --> 4"
}
ok: [localhost] => (item={u'key': u'five', u'value': 5}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "five", 
        "value": 5
    }, 
    "msg": "five --> 5"
}
ok: [localhost] => (item={u'key': u'six', u'value': 6}) => {
    "_ansible_item_result": true, 
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "item": {
        "key": "six", 
        "value": 6
    }, 
    "msg": "six --> 6"
}

TASK [debug results] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "_ansible_no_log": false, 
    "_ansible_verbose_always": true, 
    "changed": false, 
    "results": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "one", 
                    "value": 1
                }, 
                "msg": "one --> 1"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "two", 
                    "value": 2
                }, 
                "msg": "two --> 2"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "three", 
                    "value": 3
                }, 
                "msg": "three --> 3"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "four", 
                    "value": 4
                }, 
                "msg": "four --> 4"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "five", 
                    "value": 5
                }, 
                "msg": "five --> 5"
            }, 
            {
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "_ansible_verbose_always": true, 
                "item": {
                    "key": "six", 
                    "value": 6
                }, 
                "msg": "six --> 6"
            }
        ]
    }
}

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0   
```
https://gist.github.com/alikins/e5547538837c403c26f31ba5707c97d2  is a diff that prints out the order the items are ran in and the results order, and a sample section where the stdout debug output looks wrong but everywhere else shows it correctly.
@dim-on-github @zeelax @ahelal @rnurgaliyev  Are you running with the default callback? A different shipped callback or a custom callback?
@alikins I am using standard callback for display. I have other callback but for logging. 
So after much debugging we concluded this is mostly a display issue, task items are getting executed in the correct order and reported back to the main thread also in the correct order. 

This does not mean this is not a bug, but you can be assured that the actions are being executed in the order expected.

Resolving patch:
diff --git a/lib/ansible/plugins/strategy/__init__.py b/lib/ansible/plugins/strategy/__init__.py
--- a/lib/ansible/plugins/strategy/__init__.py
+++ b/lib/ansible/plugins/strategy/__init__.py
@@ -327,7 +327,7 @@ def parent_handler_match(target_handler, handler_name):
         while True:
             try:
                 self._results_lock.acquire()
-                task_result = self._results.pop()
+                task_result = self._results.popleft()
             except IndexError:
                 break
             finally:


Test patch: