Repository: hashicorp/vagrant (Ruby)

Issue:
Vagrant gathered an unknown Ansible version: and falls back on the compatibility mode '1.8'
### Debug output

https://gist.github.com/astridx/3d28bfeb109fb72db844083c8bb0c811

### Expected behavior

Use Ansible version 2.14.6.

```
$ ansible --version
ansible [core 2.14.6]
  config file = /home/me/git/ansible_me.de/ansible.cfg
  configured module search path = ['/home/me/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/me/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

```


### Actual behavior

```
==> bookworm: Running provisioner: ansible...
Vagrant gathered an unknown Ansible version:


and falls back on the compatibility mode '1.8'.

```

### Reproduction information 

#### Vagrant version

```
 vagrant -v
Vagrant 2.3.7
```


#### Host operating system

```
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

```

#### Guest operating system

```
$ vagrant ssh
Linux bookworm 6.1.0-9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jul 20 07:37:27 2023 from 192.168.121.1
vagrant@bookworm:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm
```


#### Vagrantfile

```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  if Vagrant.has_plugin?("vagrant-env") then
    config.env.enable
  end
  memory = ENV["VAGRANT_MEMORY"] || 8192
  box_name = ENV["VAGRANT_NAME"] || "bookworm"
  box_os = ENV["VAGRANT_OS"] || "debian/bookworm64"

  # Apache webserver
  config.vm.network "forwarded_port", guest: 80, host: 8089
  config.vm.network "forwarded_port", guest: 443, host: 8443

  # Never sync the current directory to /vagrant.
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.provider "libvirt" do |lv, override|
    lv.memory = memory
    lv.nested = true
  end

  config.vm.define box_name, primary: true do |sub|
      sub.vm.box = box_os
      sub.vm.provision :ansible do |s|
        s.playbook = "bootstrap.yml"
      end
  end
end

```

It is the same if I add the option `compatibility_mode = "2.0"`

```
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  if Vagrant.has_plugin?("vagrant-env") then
    config.env.enable
  end
  memory = ENV["VAGRANT_MEMORY"] || 8192
  box_name = ENV["VAGRANT_NAME"] || "bookworm"
  box_os = ENV["VAGRANT_OS"] || "debian/bookworm64"

  # Apache webserver
  config.vm.network "forwarded_port", guest: 80, host: 8089
  config.vm.network "forwarded_port", guest: 443, host: 8443

  # Never sync the current directory to /vagrant.
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.provider "libvirt" do |lv, override|
    lv.memory = memory
    lv.nested = true
  end

  config.vm.define box_name, primary: true do |sub|
      sub.vm.box = box_os
      sub.vm.provision :ansible do |s|
        s.compatibility_mode = "2.0"
        s.playbook = "bootstrap.yml"
      end
  end
end

```


Discussion:
I have the same issue.
Here is my environment:
  - ansible-core version: 2.12.5
  - Host OS: Ubuntu 20.04.6 LTS
  - Guest OS: Ubuntu 20.04.5 LTS

```shell
$ ansible --version
ansible [core 2.12.5]
  config file = /home/ikp/workspace/aic/ai_room/ansible.cfg
  configured module search path = ['/home/ikp/workspace/aic/ai_room/kubespray/library']
  ansible python module location = /home/ikp/workspace/aic/ai_room/.venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/ikp/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ikp/workspace/aic/ai_room/.venv/bin/ansible
  python version = 3.11.2 (main, Feb 27 2023, 06:21:32) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

$ vagrant ssh aic-1 -c "lsb_release -a"
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal
```

Here is my debug log (provisioning part only): https://gist.github.com/190ikp/168e3baa6484857e43d1e63bd631e073

I assume that the following lines are the cause of the issue ([L1454-L1462](https://gist.github.com/190ikp/168e3baa6484857e43d1e63bd631e073#file-provision-log-L1454-L1462)):
```shell
 INFO subprocess: Starting process: ["/home/ikp/workspace/aic/ai_room/.venv/bin/python3", "-c", "\"import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))\""]
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO interface: warn: Vagrant gathered an unknown Ansible version:


and falls back on the compatibility mode '1.8'.
```

I think this script should return the version of ansible-core.

```shell
$ python3 -c "import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))"
ansible 5.7.1

$ python3 -c "import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible-core'))"
ansible 2.12.5
```

I am not familiar with Ruby or the difference between Ansible and Ansible Core, so I am not confident that I can fix it correctly. But I think it can be resolved by rewriting it to branch between Ansible versions 2.10 and 2.11.
I found there are three cases to switch:
- ansible < 3: use `importlib.metadata.version('ansible')`
- ansible >= 3 and < 4: use `importlib.metadata.version('ansible-base')`
- ansible >= 4: use `importlib.metadata.version('ansible-core')`

or simply fix by allowing ansible major version > 2.

---
> It is the same if I add the option compatibility_mode = "2.0"

My Vagrantfile can fall back to 2.0 mode by adding `ansible.compatibility_mode = "2.0"`, so I have no idea how to fix this issue.

---

NOTE: I finally found Vagrant <= 2.3.4 can avoid this issue without compatibility_mode = "2.0".
I have this issue too.

I created a GitHub repo which can be used to recreate the environment generating the problem.
[pjgoodall/vagrant-ansible-provision](https://github.com/pjgoodall/vagrant-ansible-provision)

Parentage of the environment is in [README.md](https://github.com/hashicorp/vagrant/files/12533921/README.md)

Also experiencing issues with the detected ansible version. Last working vagrant version is `2.3.4` on MacOS
with ansible. With later versions it falls back to `1.8` mode. 

```
Python 3.9.16 (main, Jun  6 2023, 11:35:35)
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
>>> import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))
ansible 7.7.0

import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible-core'))
ansible 2.14.8

```

The following merged PR has changed from `2.3.4` to later versions;
https://github.com/hashicorp/vagrant/pull/13054 Since this change it no longer seems to be working.
I'm failing to understand if this issue is being worked on.  #13054 is marked as closed. 
How can it be closed if it is still broken? I'm trying to work through some introductory exercises on provisioning. 

On Ubuntu 22.04.3 LTS the following prevents the error. Obviously not a fix.

```
sudo apt install vagrant=2.3.4
sudo apt-mark hold vagrant=2.3.4
```

Still failing on 2.3.7-1
```
apt list -a vagrant
Listing... Done
vagrant/jammy,now 2.3.7-1 amd64 [installed]
```
```
==> server: Running provisioner: ansible...
Vagrant gathered an unknown Ansible version:


and falls back on the compatibility mode '1.8'.

Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
```
I can confirm this happening on 2.3.7. The host is FreeBSD 13.2-RELEASE-p3 GENERIC but I don't think it is relevant.

I see from the previously mentioned PR #13054, that the discovery method changed to calling `python3 -c "...."`.
This command does return successfully when ran manually on the host and it reports:
```
$ python3 -c "import importlib.metadata; print('ansible '+importlib.metadata.version('ansible'))"
ansible 8.2.0
```
What *I think* happens in this case, also based on @190ikp 's output, is that the subprocess ran from Vagrant either fails or it doesn't return stdout correctly and that silently propagates to Vagrant as no Ansible version detected and falling back to the safe compatibility version.

I do see proper version checking as [part of testing](https://github.com/hashicorp/vagrant/blob/ba10fe9ee6f4c0d58deb51503c36c69b4d9442bb/test/unit/plugins/provisioners/ansible/provisioner_test.rb#L327) so this further backs up my assumption that this issue is with the subprocess call.

Indeed explicitly setting `compatibility_mode = "2.0"` works around the issue.
I believe it's quoted incorrectly.  In `plugins/provisioners/ansible/provisioner/guest.rb` the command is passed as a string for a shell to take apart, so the `\"`s are needed to pass quotes which are removed before the argument is provided to the Python interpreter, whereas in `host.rb` the arguments are already split into an list, and so the quotes make it to the Python interpreter, which interprets the command *not* as

```python
import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))
```

but as

```python
"import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))"
```

a string literal, which generates no output when evaluated non-interactively.

Removing those quotes fixed it for me.

```diff
diff --git a/plugins/provisioners/ansible/provisioner/host.rb b/plugins/provisioners/ansible/provisioner/host.rb
index ce3752f40..9f9dc5d0d 100644
--- a/plugins/provisioners/ansible/provisioner/host.rb
+++ b/plugins/provisioners/ansible/provisioner/host.rb
@@ -113,7 +113,7 @@ module VagrantPlugins
         def gather_ansible_version
           raw_output = ''
           command = ['python3', '-c',
-                     "\"import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))\""]
+                     "import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))"]
 
           command << {
             notify: [:stdout, :stderr]
```
@quartsize tested the change locally and works like a charm!
@soapy1 @chrisroberts could you please try the patch from https://github.com/hashicorp/vagrant/issues/13234#issuecomment-1789452544? I'd send a PR, but due to the license change, I am no longer legally allowed to :-(
I ran into the same problem today using Fedora Workstation as the host and Ubuntu Jammy as the guest. The patch from @quartsize worked. 

It looks like this just needs a PR? 



Patch:
diff --git a/plugins/provisioners/ansible/provisioner/host.rb b/plugins/provisioners/ansible/provisioner/host.rb
index ce3752f409d..9f9dc5d0d59 100644
--- a/plugins/provisioners/ansible/provisioner/host.rb
+++ b/plugins/provisioners/ansible/provisioner/host.rb
@@ -113,7 +113,7 @@ def execute_command_from_host(command)
         def gather_ansible_version
           raw_output = ''
           command = ['python3', '-c',
-                     "\"import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))\""]
+                     "import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))"]
 
           command << {
             notify: [:stdout, :stderr]


Test patch:
diff --git a/test/unit/plugins/provisioners/ansible/provisioner_test.rb b/test/unit/plugins/provisioners/ansible/provisioner_test.rb
index c6566d2d433..b4466d21ee6 100644
--- a/test/unit/plugins/provisioners/ansible/provisioner_test.rb
+++ b/test/unit/plugins/provisioners/ansible/provisioner_test.rb
@@ -77,7 +77,7 @@ def find_last_argument_after(ref_index, ansible_playbook_args, arg_pattern)
   def self.it_should_check_ansible_version
     it "execute 'Python ansible version check before executing 'ansible-playbook'" do
       expect(Vagrant::Util::Subprocess).to receive(:execute)
-        .once.with('python3', '-c', "\"import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))\"", { notify: %i[
+        .once.with('python3', '-c', "import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))", { notify: %i[
                      stdout stderr
                    ] })
       expect(Vagrant::Util::Subprocess).to receive(:execute)
@@ -1054,7 +1054,7 @@ def ensure_that_config_is_valid
         expect(Vagrant::Util::Subprocess).to receive(:execute)
           .once
           .with('python3', '-c',
-                "\"import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))\"", { notify: %i[stdout stderr] })
+                "import importlib.metadata; print('ansible ' + importlib.metadata.version('ansible'))", { notify: %i[stdout stderr] })
           .and_return(default_execute_result)
         expect(Vagrant::Util::Subprocess).to receive(:execute)
           .once