internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ansible/ansible

Issue:
ansible_facts no longer passed to templates
I note Ansible is no longer passing facts to templates, at least it doesn't work for me on -devel (last pulled a few minutes ago). I've looked at the code, in `lib/ansible/runner/__init__.py` and I believe it's due to `execute_template()` not calling `_add_result_to_setup_cache()` but I don't have a clue when or how to invoke the latter.

Here's a test run:

```
$ ANSIBLE_SSH_ARGS="" ANSIBLE_REMOTE_USER=f2 ansible-playbook --connection=ssh j1.yaml -v

PLAY [a1.ww.mens.de] ********************* 

GATHERING FACTS ********************* 
ok: [a1.ww.mens.de]

TASK: [JPSETUP] ********************* 
ok: [a1.ww.mens.de] => {"ansible_facts": {"users_logged_in": 3}}

TASK: [Demotemplate] ********************* 
ok: [a1.ww.mens.de] => {"changed": false, "dest": "/tmp/info.txt", "group": "adm", "md5sum": "30553208a6f542667c47e728e9bb7f9a", "mode": "0644", "path": "/tmp/info.txt", "src": "/home/f2/.ansible/tmp/ansible-1342596365.48-141559060171382/source", "state": "file", "user": "f2"}

PLAY RECAP ********************* 
a1.ww.mens.de                  : ok=3    changed=1    unreachable=0    failed=0    
```

Playbook:

```

---
- hosts: a1.ww.mens.de
  vars:
     name: v2
  tasks:
  - name: JPSETUP
    action: jpsetup
  - name: Demotemplate
    action: template src=info.in dest=/tmp/info.txt
```

Facts 

```
$ cat library/jpsetup 
#!/bin/sh

COUNT=`who | wc -l`
cat <<EOF
{
   "ansible_facts" : {
      "users_logged_in" : $COUNT
   }
}
EOF
```

Template

```
$ cat info.in
version: {{ name }}
logged-in: {{ users_logged_in }}
logged-in: ${users_logged_in}
```

Output

```
$ cat /tmp/info.txt
version: v2
logged-in: 
logged-in: ${users_logged_in}
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/playbook/__init__.py b/lib/ansible/playbook/__init__.py
--- a/lib/ansible/playbook/__init__.py
+++ b/lib/ansible/playbook/__init__.py
@@ -221,7 +221,7 @@ def _run_task(self, play, task, is_handler):
 
         # add facts to the global setup cache
         for host, result in results['contacted'].iteritems():
-            facts = results.get('ansible_facts', {})
+            facts = result.get('ansible_facts', {})
             self.SETUP_CACHE[host].update(facts)
 
         self.stats.compute(results)


Test patch:
Repository: ansible/ansible

Issue:
copy chains to file incorrectly (mode= is incorrect)
```
$ rm /tmp/ff1
$ ansible 127.0.0.1 -v -c local -m copy -a 'src=JP dest=/tmp/ff1 mode=0600'
127.0.0.1 | success >> {
    "changed": true, 
    "dest": "/tmp/ff1", 
    "group": "wheel", 
    "md5sum": "f805a1c8bd9e3adec9b6a94844d00941", 
    "mode": "0644", 
    "module": "copy", 
    "path": "/tmp/ff1", 
    "src": "/Users/jpm/.ansible/tmp/ansible-1343138359.16-236375388664908/JP", 
    "state": "file", 
    "user": "jpm"
}
$ ls -l /tmp/ff1
-rw-r--r--  1 jpm  wheel  33 Jul 24 15:59 /tmp/ff1
```

@mpdehaan said:

> > you will probably need to debug around the 'results2' stuff in Runner/init.py to see what the file module is being called with and seeing what it returns. It may there is just a problem in the file module that can be found with hacking/test-module Note that hacking/test-module does not understand daisy chaining



Discussion:
in execute_copy try changing:

```
  self.module_args = "src=%s dest=%s" % (tmp_src, dest)
        return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
```

to

```
        return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
```

and see if that fixes it.   Looks like it is internally throwing away other arguments.   This is probably also true of other places where we do daisychain('file')


Resolving patch:
diff --git a/lib/ansible/runner/__init__.py b/lib/ansible/runner/__init__.py
--- a/lib/ansible/runner/__init__.py
+++ b/lib/ansible/runner/__init__.py
@@ -317,7 +317,6 @@ def _execute_copy(self, conn, tmp, inject=None):
             conn.put_file(source, tmp_src)
 
             # run the copy module
-            self.module_args = "src=%s dest=%s" % (tmp_src, dest)
             return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
 
         else:


Test patch:
Repository: ansible/ansible

Issue:
Implement 'polite' YAML validation
Typos happen.

Typos in YAML can be especially difficult to figure out, and they currently present themselves as enormous stack trace vomits [1]

Implement something friendlier to handle these situations. Try starting by attempting to port the yaml validation method from Taboot [2].

[1] http://fpaste.org/3cWY/
[2] https://github.com/tbielawa/Taboot/blob/master/taboot/scripts.py#L92



Discussion:

Resolving patch:
diff --git a/lib/ansible/errors.py b/lib/ansible/errors.py
--- a/lib/ansible/errors.py
+++ b/lib/ansible/errors.py
@@ -30,3 +30,5 @@ class AnsibleFileNotFound(AnsibleError):
 class AnsibleConnectionFailed(AnsibleError):
     pass
 
+class AnsibleYAMLValidationFailed(AnsibleError):
+    pass
diff --git a/lib/ansible/utils.py b/lib/ansible/utils.py
--- a/lib/ansible/utils.py
+++ b/lib/ansible/utils.py
@@ -229,9 +229,29 @@ def parse_yaml_from_file(path):
 
     try:
         data = file(path).read()
+        return parse_yaml(data)
     except IOError:
         raise errors.AnsibleError("file not found: %s" % path)
-    return parse_yaml(data)
+    except yaml.YAMLError, exc:
+        if hasattr(exc, 'problem_mark'):
+            mark = exc.problem_mark
+            if mark.line -1 >= 0:
+                before_probline = data.split("\n")[mark.line-1]
+            else:
+                before_probline = ''
+            probline = data.split("\n")[mark.line]
+            arrow = " " * mark.column + "^"
+            msg = """Syntax Error while loading YAML script, %s
+Note: The error may actually appear before this position: line %s, column %s
+
+%s
+%s
+%s""" % (path, mark.line + 1, mark.column + 1, before_probline, probline, arrow)
+        else:
+            # No problem markers means we have to throw a generic
+            # "stuff messed up" type message. Sry bud.
+            msg = "Could not parse YAML. Check over %s again." % path
+        raise errors.AnsibleYAMLValidationFailed(msg)
 
 def parse_kv(args):
     ''' convert a string of key/value items to a dict '''


Test patch:
Repository: ansible/ansible

Issue:
--version should output the git date if running from checkout
this means any sort of make install really needs to drop in some kind of date file???



Discussion:
done


Resolving patch:
diff --git a/lib/ansible/utils.py b/lib/ansible/utils.py
--- a/lib/ansible/utils.py
+++ b/lib/ansible/utils.py
@@ -28,6 +28,7 @@
 from ansible import color
 from ansible import __version__
 import ansible.constants as C
+import time
 
 try:
     import json
@@ -288,16 +289,19 @@ def default(value, function):
     return value
 
 def _gitinfo():
-    ''' returns a string containing git branch and commit id 
-    using gitpython if installed and native file operations if not '''
+    ''' returns a string containing git branch, commit id and commit date '''
     result = None
     repo_path = os.path.join(os.path.dirname(__file__), '..', '..', '.git')
     if os.path.exists(repo_path):
         with open(os.path.join(repo_path, "HEAD")) as f:
             branch = f.readline().split('/')[-1].rstrip("\n")
-        with open(os.path.join(repo_path, "refs", "heads", branch)) as f:
+        branch_path = os.path.join(repo_path, "refs", "heads", branch)
+        with open(branch_path) as f:
             commit = f.readline()[:10] 
-        result = "({0} {1})".format(branch, commit)
+        date = time.localtime(os.stat(branch_path).st_mtime)
+        offset = time.timezone if (time.daylight == 0) else time.altzone
+        result = "({0}) [{1}] {2} ({3:+04d})".format(branch, commit, 
+            time.strftime("%Y%m%d-%H%M%S", date), offset / -36)
     return result
 
 def version(prog):


Test patch:
Repository: ansible/ansible

Issue:
Feature: Please add fromjson() filter for Jinja2: patch inline
I'm looking at register variables, and would like to propose a one-line change to `lib/ansible/utils.py` to support JSON output.
### The patch

```
diff --git a/lib/ansible/utils.py b/lib/ansible/utils.py
index 65bc704..4b5461e 100644
--- a/lib/ansible/utils.py
+++ b/lib/ansible/utils.py
@@ -242,6 +242,7 @@ def template_from_file(basedir, path, vars):

     environment = jinja2.Environment(loader=jinja2.FileSystemLoader(basedir), trim_blocks=False)
     environment.filters['tojson'] = json.dumps
+    environment.filters['fromjson'] = json.loads
     data = codecs.open(path_dwim(basedir, path), encoding="utf8").read()
     t = environment.from_string(data)
     vars = vars.copy()
```

This patch allows us to return JSON into a register-variable:
### Playbook

```
  - name: Run jpprog.sh
    action: command ./jpprog.sh
    register: v

  - name: Populate template
    action: template src=${filename} dest=/tmp/out
```
### jpprog.sh

```

#!/bin/sh

cat <<!
{
    "number": 18,
        "name": "john"
}
!
exit 0
```
### Template

```
{% set t = v.stdout|fromjson %}

JSON struct T: {{ t }}

name = {{ t.name }} 
number = {{ t['number'] }} 
```
### Output

```
JSON struct T: {u'number': 18, u'name': u'john'}

name = john 
number = 18 
```

Good idea? :-)



Discussion:
If v.stdout returns JSON, you basically could run the script as a module.   Though that's totally fine if you don't.  I need to see about exposing other things to Jinja, like getattr as you mentioned elsewhere.

I'll accept this, send me a PR -- though I think both fromjson and tojson should be from_json and to_json to be consistent with the rest of the API.


Resolving patch:
diff --git a/lib/ansible/utils.py b/lib/ansible/utils.py
--- a/lib/ansible/utils.py
+++ b/lib/ansible/utils.py
@@ -241,7 +241,8 @@ def template_from_file(basedir, path, vars):
     ''' run a file through the templating engine '''
 
     environment = jinja2.Environment(loader=jinja2.FileSystemLoader(basedir), trim_blocks=False)
-    environment.filters['tojson'] = json.dumps
+    environment.filters['to_json'] = json.dumps
+    environment.filters['from_json'] = json.loads
     data = codecs.open(path_dwim(basedir, path), encoding="utf8").read()
     t = environment.from_string(data)
     vars = vars.copy()


Test patch:
Repository: ansible/ansible

Issue:
Change in gather_facts not working in all cases
The change from #1201 does not work in all cases because the test to see if the setup module was already run (once) is making incorrect assumptions which fail in our use-case.

What we do for provisioning is summarized as:
- Disable fact_gathering while 'locally' running:
  - Running network_facts (from DNS/yaml)
  - Running ilo or esx facts (from iLO device/ESX server)
  - Create boot-media using the above facts
  - Boot the physical/virtual system using this boot-media
- Wait for the system to come online in post-install mode
- Start the actual provisioning

This fails for the current implementation because when we start the actual provisioning, the setup module is not run for the first time because our other facts-modules already added facts to the **SETUP_CACHE**. What I would propose is that each facts-module in fact adds it's own fact to track whether it has already run. This is useful even outside the core usage, because I currently already check whether one of the facts-modules has run by testing one of the facts the module provides. Of course it would be better to not test a generic fact that we "know" has been set, but rather make this part of the standard.

So I propose every facts-module (setup included) to set the:

```
module_name: true
```

So in our case, I would have the following set:

```
module_ilo: true
module_esx: true
module_network: true
```

and the setup module would check if **module_setup** is set in order to decide whether it should run.



Discussion:
Including @dhozac into the discussion


Resolving patch:
diff --git a/lib/ansible/playbook/__init__.py b/lib/ansible/playbook/__init__.py
--- a/lib/ansible/playbook/__init__.py
+++ b/lib/ansible/playbook/__init__.py
@@ -317,7 +317,7 @@ def _do_setup_step(self, play):
         if play.gather_facts is False:
             return {}
         elif play.gather_facts is None:
-            host_list = [h for h in host_list if h not in self.SETUP_CACHE]
+            host_list = [h for h in host_list if h not in self.SETUP_CACHE or 'module_setup' not in self.SETUP_CACHE[h]]
             if len(host_list) == 0:
                 return {}
 
@@ -340,6 +340,7 @@ def _do_setup_step(self, play):
         # let runner template out future commands
         setup_ok = setup_results.get('contacted', {})
         for (host, result) in setup_ok.iteritems():
+            self.SETUP_CACHE[host].update({'module_setup': True})
             self.SETUP_CACHE[host].update(result.get('ansible_facts', {}))
         return setup_results
 


Test patch:
Repository: ansible/ansible

Issue:
"serial" can't be passed as a command line argument (-e) to ansible-playbook
``` shell
[root@cobbler ~]# cat foo.yml

---
- hosts: app-pool
  serial: ${servar}

  tasks:

    - name: write a file
      action: shell touch /tmp/foo

    - name: delete it
      action: shell rm -f /tmp/foo

[root@cobbler ~]# ansible-playbook foo.yml -e "servar=1"

PLAY [app-pool] *********************

GATHERING FACTS *********************
ok: [app02.foo.bar.com]
ok: [app03.foo.bar.com]
ok: [app05.foo.bar.com]
ok: [app04.foo.bar.com]
ok: [app01.foo.bar.com]
ok: [app06.foo.bar.com]
ok: [app07.foo.bar.com]
ok: [app08.foo.bar.com]
Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 164, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/bin/ansible-playbook", line 135, in main
    pb.run()
  File "/usr/lib/python2.6/site-packages/ansible/playbook/__init__.py", line 193, in run
    if not self._run_play(play):
  File "/usr/lib/python2.6/site-packages/ansible/playbook/__init__.py", line 378, in _run_play
    for x in range(play.serial):
TypeError: range() integer end argument expected, got str.
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/playbook/play.py b/lib/ansible/playbook/play.py
--- a/lib/ansible/playbook/play.py
+++ b/lib/ansible/playbook/play.py
@@ -75,7 +75,7 @@ def __init__(self, playbook, ds, basedir):
         self.transport    = ds.get('connection', self.playbook.transport)
         self.tags         = ds.get('tags', None)
         self.gather_facts = ds.get('gather_facts', None)
-        self.serial       = ds.get('serial', 0)
+        self.serial       = int(utils.template_ds(basedir, ds.get('serial', 0), self.vars))
 
         if isinstance(self.remote_port, basestring):
             self.remote_port = utils.template(basedir, self.remote_port, self.vars)


Test patch:
Repository: ansible/ansible

Issue:
inline comment on hosts line causes traceback
Given a hosts line like this:-

  localhost  # this is the localhost

then ansible will bomb with the following traceback

Traceback (most recent call last):
  File "/Users/nigel/Work/ansible/ansible/bin/ansible", line 157, in <module>
    (runner, results) = cli.run(options, args)
  File "/Users/nigel/Work/ansible/ansible/bin/ansible", line 83, in run
    inventory_manager = inventory.Inventory(options.inventory)
  File "/Users/nigel/Work/ansible/ansible/lib/ansible/inventory/**init**.py", line 91, in **init**
    self.parser = InventoryParser(filename=host_list)
  File "/Users/nigel/Work/ansible/ansible/lib/ansible/inventory/ini.py", line 40, in **init**
    self._parse()
  File "/Users/nigel/Work/ansible/ansible/lib/ansible/inventory/ini.py", line 44, in _parse
    self._parse_base_groups()
  File "/Users/nigel/Work/ansible/ansible/lib/ansible/inventory/ini.py", line 112, in _parse_base_groups
    (k,v) = t.split("=")
ValueError: need more than 1 value to unpack



Discussion:
Am working on this, will attach patch later


Resolving patch:
diff --git a/lib/ansible/inventory/ini.py b/lib/ansible/inventory/ini.py
--- a/lib/ansible/inventory/ini.py
+++ b/lib/ansible/inventory/ini.py
@@ -109,6 +109,8 @@ def _parse_base_groups(self):
                         self.hosts[hn] = host
                     if len(tokens) > 1:
                         for t in tokens[1:]:
+                            if t.startswith('#'):
+                                break
                             (k,v) = t.split("=")
                             host.set_variable(k,v)
                     self.groups[active_group_name].add_host(host)


Test patch:
Repository: ansible/ansible

Issue:
When using group_by, invalidate the group variables cache to force variable loads
Suppose we use group_by to create groups based on OS distribution.   It was reported that in order to assign variables based on distribution we have to have the group name already in inventory for group_vars to be loaded.

We should remove this requirement.



Discussion:
The issue is that once a host's vars have been gathered, they are cached (Inventory.get_variables). Adding a new group should invalidate the cache, and things should work as expected.

updated the description, thanks!


Resolving patch:
diff --git a/lib/ansible/runner/action_plugins/group_by.py b/lib/ansible/runner/action_plugins/group_by.py
--- a/lib/ansible/runner/action_plugins/group_by.py
+++ b/lib/ansible/runner/action_plugins/group_by.py
@@ -68,6 +68,7 @@ def run(self, conn, tmp, module_name, module_args, inject, complex_args=None, **
                 inv_group = ansible.inventory.Group(name=group)
                 inventory.add_group(inv_group)
             for host in hosts:
+                del self.runner.inventory._vars_per_host[host]
                 inv_host = inventory.get_host(host)
                 if not inv_host:
                     inv_host = ansible.inventory.Host(name=host)


Test patch:
Repository: ansible/ansible

Issue:
remote_tmp is ignored when using a sudo user other than 'root'.
Running ansible -U otheruser will cause the `remote_tmp` to always be set as /tmp 

I have a world-writeable directory set as my remote_tmp which gets overridden by this behavior.  In my environment /tmp sometimes fills up faster than my world-writeable dir, so I'd like the behavior to specify an explicit remote tmp directory.

Thank you!



Discussion:
bcoca identified the logic at lib/ansible/runner/**init**.py line 575

https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/__init__.py#L576


Resolving patch:
diff --git a/lib/ansible/runner/__init__.py b/lib/ansible/runner/__init__.py
--- a/lib/ansible/runner/__init__.py
+++ b/lib/ansible/runner/__init__.py
@@ -573,7 +573,7 @@ def _make_tmp_path(self, conn):
 
         basefile = 'ansible-%s-%s' % (time.time(), random.randint(0, 2**48))
         basetmp = os.path.join(C.DEFAULT_REMOTE_TMP, basefile)
-        if self.sudo and self.sudo_user != 'root':
+        if self.sudo and self.sudo_user != 'root' and basetmp.startswith('$HOME'):
             basetmp = os.path.join('/tmp', basefile)
 
         cmd = 'mkdir -p %s' % basetmp


Test patch:
Repository: ansible/ansible

Issue:
Ansible breaks with modules that contain non-ASCII characters
I'm writing a module that includes accented chars (my name ;), but although it's UTF-8 encoded, Ansible fails to use it.

The problem is that it tries to re-encode it (in the file `lib/ansible/runner/__init__.py:209`), but what happens is that Python first decodes it as ASCII, which breaks on UTF-8 encoded modules.

Ansible breaks with modules that contain non-ASCII characters
I'm writing a module that includes accented chars (my name ;), but although it's UTF-8 encoded, Ansible fails to use it.

The problem is that it tries to re-encode it (in the file `lib/ansible/runner/__init__.py:209`), but what happens is that Python first decodes it as ASCII, which breaks on UTF-8 encoded modules.



Discussion:

Resolving patch:
diff --git a/lib/ansible/runner/__init__.py b/lib/ansible/runner/__init__.py
--- a/lib/ansible/runner/__init__.py
+++ b/lib/ansible/runner/__init__.py
@@ -206,7 +206,12 @@ def _transfer_str(self, conn, tmp, name, data):
         afd, afile = tempfile.mkstemp()
         afo = os.fdopen(afd, 'w')
         try:
-            afo.write(data.encode('utf8'))
+            if not isinstance(data, unicode):
+                #ensure the data is valid UTF-8
+                data.decode('utf-8')
+            else:
+                data = data.encode('utf-8')
+            afo.write(data)
         except:
             raise errors.AnsibleError("failure encoding into utf-8")
         afo.flush()


Test patch:
Repository: ansible/ansible

Issue:
re-enable 'force' parameter on copy
This parameter, default True, settable to False, can be used to not copy a file if it already exists.

There was an implementation problem with this option so I disabled it on the devel branch and it needs to be added back.



Discussion:

Resolving patch:
diff --git a/lib/ansible/module_common.py b/lib/ansible/module_common.py
--- a/lib/ansible/module_common.py
+++ b/lib/ansible/module_common.py
@@ -176,7 +176,9 @@ def __init__(self, argument_spec, bypass_checks=False, no_log=False,
         self.aliases = {}
         
         if add_file_common_args:
-            self.argument_spec.update(FILE_COMMON_ARGUMENTS)
+            for k, v in FILE_COMMON_ARGUMENTS.iteritems():
+                if k not in self.argument_spec:
+                    self.argument_spec[k] = v
 
         os.environ['LANG'] = MODULE_LANG
         (self.params, self.args) = self._load_params()


Test patch:
Repository: ansible/ansible

Issue:
fcntl.flock error after ansible installation under Solaris10 x86 (see google group discussion ansible project)
Solaris 10 implementation of python function fcntl.flock seems not to allow exclusive locks for files open for reading as well as shared locks for files open for writing.
Running ansible under Solaris it crashes with following traceback:
~ > ansible all -m ping
Traceback (most recent call last):
  File "/usr/gnu/64/bin/ansible", line 149, in <module>
    callbacks.display("", log_only=True)
  File "/usr/gnu/64/lib/python2.6/site-packages/ansible/callbacks.py", line 98, in display
    log_flock()
  File "/usr/gnu/64/lib/python2.6/site-packages/ansible/callbacks.py", line 79, in log_flock
    fcntl.flock(LOG_LOCK, fcntl.LOCK_EX)
IOError: [Errno 9] Bad file number

In a test program changing the type of lock requested to fcntl.LOCK_SH solved the problem.

Regards,

Karl



Discussion:

Resolving patch:
diff --git a/lib/ansible/callbacks.py b/lib/ansible/callbacks.py
--- a/lib/ansible/callbacks.py
+++ b/lib/ansible/callbacks.py
@@ -68,12 +68,9 @@ def log_lockfile():
     tempdir = tempfile.gettempdir() 
     uid = os.getuid()
     path = os.path.join(tempdir, ".ansible-lock.%s" % uid)
-    if not os.path.exists(path):
-        fh = open(path, 'w')
-        fh.close()   
     return path
 
-LOG_LOCK = open(log_lockfile(), 'r')
+LOG_LOCK = open(log_lockfile(), 'w')
 
 def log_flock():
     fcntl.flock(LOG_LOCK, fcntl.LOCK_EX)


Test patch: