Repository: canonical/cloud-init

Issue:
OVF datasource should check if instant id is still on VMware Platform
This bug was originally filed in Launchpad as [LP: #1835205](https://bugs.launchpad.net/cloud-init/+bug/1835205)
<details>
<summary>Launchpad details</summary>
<pre>
affected_projects = []
assignee = None
assignee_name = None
date_closed = None
date_created = 2019-07-03T11:24:36.299696+00:00
date_fix_committed = None
date_fix_released = None
id = 1835205
importance = medium
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1835205
milestone = None
owner = pengpengs
owner_name = Pengpeng Sun
private = False
status = triaged
submitter = pengpengs
submitter_name = Pengpeng Sun
tags = []
duplicates = []
</pre>
</details>

_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-07-03T11:24:36.299696+00:00_

Currently DatasourceOVF does not check the instance id to determine if current instance is a new instance or not, so that every booting, cloud-init will go through entire datasource list.

This leads an issue:
When a VM's network is customized to static IP by cloud-init DatasourceOVF and then the VM is rebooted, the VM's network will be changed to DHCP after rebooting since this time no customization config file will be found. cloud-init uses DatasourceNone which change network to "default" configuration.

The expected behaviors are:
1. cloud-init checks the instance id to know if this instance is "iid-vmware-xxxxxx" which means it's a VMware VM. If yes, it should always use datasourceOVF.
2. When there has customization config file, datasourceOVF parse the configures and enable nics as usual.
3. When there is no customization config file, datasourceOVF wait for it until timeout and then cloud-init does NOT apply other datasources.


Discussion:
_Launchpad user **Ryan Harper(raharper)** wrote on 2019-07-18T20:40:03.474066+00:00_

Thanks for filling a bug.

Cloud-init does check instance-id by looking in /var/lib/cloud/data/ at files 'instance-id' and 'previous-instance-id' and will check if the existing Datasource's metadata provides the same instance-id.

The behavior you describe *sounds* like the OVF datasource changed.

Please run cloud-init collect-logs as root and attach the tarball so we can see what's going on.

Thanks!
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-07-24T08:04:21.726080+00:00_

Hi Ryan,

After cloud-init customization done, I saw /var/lib/cloud/instance links to /var/lib/cloud/instances/iid-vmware-XXXXXXXX.  But this instance seems not take effect at next reboot, cloud-init always think it's a new instance.  Do a reboot here, the /var/lib/cloud/instance links to /var/lib/cloud/instances/iid-datasource-none

I think `check_instance_id` is not implemented in OVF datasource.

Thanks,
Pengpeng



_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-07-24T08:05:01.768839+00:00_

Attach logs after cloud-init customization by OVF datasource.
_Launchpad attachments_: [cloud-init-after-customization.tar.gz](https://api.launchpad.net/devel/cloud-init/+bug/1835205/+attachment/5278918/data)
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-07-24T08:05:29.344092+00:00_

Attach logs after reboot.
_Launchpad attachments_: [cloud-init-after-boot.tar.gz](https://api.launchpad.net/devel/cloud-init/+bug/1835205/+attachment/5278919/data)
_Launchpad user **Ilia(ilia-zenkevich)** wrote on 2019-07-24T13:25:31.376996+00:00_

Hi all.
Faced with the same issue.

Environment:
vCloud Director 9.1
Ubuntu Server 16.04.1 x64
open-vm-tools 2:10.2.0-3~ubuntu0.16.04.1
cloud-init 19.1-1-gbaa47854-0ubuntu1~16.04.1

cloud-init log for second and later boots always contains

2019-07-23 13:22:39,347 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File
2019-07-23 13:22:39,347 - util.py[DEBUG]: Running command ['vmware-rpctool', 'info-get guestinfo.ovfEnv'] with allowed return codes [0]
 (shell=False, capture=True)
2019-07-23 13:22:39,353 - handlers.py[DEBUG]: finish: init-local/search-OVF: FAIL: no local data found from DataSourceOVF
  File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOVF.py", line 394, in read_ovf_environment
    props = get_properties(contents)
  File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOVF.py", line 553, in get_properties
    raise XmlError("No 'PropertySection's")
cloudinit.sources.DataSourceOVF.XmlError: No 'PropertySection's


If I run "vmware-rpctool info-get guestinfo.ovfEnv" manually, I can't see "PropertySection" sections, but the question is: Is this section something that have to be in the response?

_Launchpad user **Ryan Harper(raharper)** wrote on 2019-07-24T14:34:55.308416+00:00_

Note that check_instance_id() is only needed if you want to avoid fetching metadata/userdata a second time; it does not prevent cloud-init from knowing it's on the same instance; In all cases, the metadata that indicates the instance-id _must_ remain associated with the instance.


Looking at the logs; it appears that after reboot the Customization file is not found and OVF datasource _get_data() method returns False; which tells cloud-init that it's not using OVF as a datasource.

1. firstboot finds OVF Customication 

2019-07-24 07:50:43,682 - util.py[DEBUG]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init-local' at Wed, 24 Jul 2019 07:50:43 +0000. Up 12.30 seconds.
...
2019-07-24 07:50:43,802 - __init__.py[DEBUG]: Searching for local data source in: ['DataSourceOVF']
2019-07-24 07:50:43,802 - handlers.py[DEBUG]: start: init-local/search-OVF: searching for local data from DataSourceOVF
2019-07-24 07:50:43,802 - __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceOVF.DataSourceOVF'>
2019-07-24 07:50:43,802 - __init__.py[DEBUG]: Update datasource metadata and network config due to events: New instance first boot

2019-07-24 07:50:43,813 - DataSourceOVF.py[DEBUG]: VMware Virtualization Platform found
2019-07-24 07:50:43,813 - DataSourceOVF.py[DEBUG]: Found the customization plugin at /usr/lib/open-vm-tools/plugins/vmsvc/libdeployPkgPlugin.so
2019-07-24 07:50:43,813 - DataSourceOVF.py[DEBUG]: Waiting for VMware Customization Config File
2019-07-24 07:50:48,818 - util.py[DEBUG]: waiting for configuration file took 5.005 seconds
2019-07-24 07:50:48,823 - DataSourceOVF.py[DEBUG]: Found VMware Customization Config File at /var/run/vmware-imc/cust.cfg
2019-07-24 07:50:49,928 - handlers.py[DEBUG]: finish: init-local/search-OVF: SUCCESS: found local data from DataSourceOVF
2019-07-24 07:50:49,929 - stages.py[INFO]: Loaded datasource DataSourceOVF - DataSourceOVF [seed=vmware-tools]

This sets the instance id to: 

2019-07-24 07:50:49,945 - util.py[DEBUG]: Creating symbolic link from '/var/lib/cloud/instance' => '/var/lib/cloud/instances/iid-vmware-QokNwWPI'


2. Second boot of same instance does not find metadata from DataSourceOVF

2019-07-24 07:56:49,746 - DataSourceOVF.py[DEBUG]: VMware Virtualization Platform found
2019-07-24 07:56:49,746 - DataSourceOVF.py[DEBUG]: Found the customization plugin at /usr/lib/open-vm-tools/plugins/vmsvc/libdeployPkgPlugin.so
2019-07-24 07:56:49,746 - DataSourceOVF.py[DEBUG]: Waiting for VMware Customization Config File

2019-07-24 07:58:19,856 - util.py[DEBUG]: waiting for configuration file took 90.110 seconds
2019-07-24 07:58:19,858 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File

2019-07-24 07:58:20,152 - main.py[DEBUG]: [local] Exiting without datasource


Why did the customization data for the instance go away?

_Launchpad user **Ilia(ilia-zenkevich)** wrote on 2019-07-24T16:47:24.325317+00:00_

Because cloud-init deletes it in DataSourceOVF.py
_Launchpad user **Ryan Harper(raharper)** wrote on 2019-07-24T17:48:00.227924+00:00_

You're quite right.  I don't understand how this is expected to work.

Removing the customization file is going to prevent cloud-init from knowing it's on the same instance if the instance-id came from it.

but I don't think it did, I see this:



 def read_vmware_imc(config):
@@ -296,6 +330,9 @@ def read_vmware_imc(config):
     if config.timezone:
         cfg['timezone'] = config.timezone
 
+    # Generate a unique instance-id so that re-customization will
+    # happen in cloud-init
+    md['instance-id'] = "iid-vmware-" + util.rand_str(strlen=8)


Which is completely broken w.r.t keeping the same datasource.  The comment suggests that when redeploying to the same instance, "new" customization didn't happen, so to work around that
the datasource generates a new "random" instance-id, to avoid cloud-init's check on instance-id.

The OVF datasource is going to need quite a bit of work to sort out which paths are working and which are not; having the platform provide a consistent set of metadata to the instance in a reliable form for the life-time of the instance is a requirement.  This doesn't appear to be happening via the Imc path.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-07-25T03:01:01.924285+00:00_

Hi Ryan,

Thanks for the investigation.

There is no consistent user data provided to OVF datasource, the "VMware Customization Config File" is available only when the customization happens on the VM. 
1. Do Guest OS Customization on the VM (firstboot finds OVF Customization)
With customization happened on VM, Open-VM-Tools can find and then copy cust.cfg from ESX server to the VM path /var/run/vmware-imc/, OVF Datasource will find the /var/run/vmware-imc/cust.cfg and then load it.
> 2019-07-24 07:50:48,823 - DataSourceOVF.py[DEBUG]: Found VMware Customization Config File at /var/run/vmware-imc/cust.cfg

2. Reboot VM (Second boot of same instance does not find metadata from DataSourceOVF)
This time without customization happened on VM, so no cust.cfg will be found by Open-VM-Tools, OVF Datasource can not find /var/run/vmware-imc/cust.cfg, and then existing without datasource.
> 2019-07-24 07:58:19,856 - util.py[DEBUG]: waiting for configuration file took 90.110 seconds
> 2019-07-24 07:58:19,858 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File

As far as I know, there are 2 paths.
1. customize VM by cloud-init and then re-customize VM by cloud-init. 
This path works now since "random" instance-id keep cloud-init to search userdata for OVF datasource every boot

2. customization VM by cloud-int and then just reboot
This path is broken, the expected behavior is at the second reboot, cloud-init keep network configuration unchanged. 
I do not think up a good solution, please shed some light on how to achieve it.


_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-07-25T03:03:55.076063+00:00_

Links to VMware bug# 2377600
_Launchpad user **Ilia(ilia-zenkevich)** wrote on 2019-08-01T10:16:59.223605+00:00_

What does it mean? "Links to VMware bug# 2377600" - can't find bug with such id. Can you please give a direct link?
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-08-01T10:34:24.327741+00:00_

@ilia, It should be "Links to VMware INTERNAL bug# 2377600". It just a tracker for me.
_Launchpad user **Eduardo Otubo(otubo)** wrote on 2019-08-16T11:43:26.287500+00:00_

Blocks Red Hat bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1593010
_Launchpad user **Eduardo Otubo(otubo)** wrote on 2019-08-16T11:58:57.806093+00:00_

@Pengpeng Sun do you have a status on that? Would another pair of hands be useful? I run into this bug while checking the above linked BZ. I have access to vSphere and could help taking a look.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-08-19T09:00:19.554168+00:00_

@otubo, From the Red Hat bug, I think it's not quite same with this one. For this one is after enabling VMware guest customization with cloud-init. With the latest cloud-init code, cloud-init will not disable it self due to VMware guest customization is enabled. 

For Red Hat Linux, I filed a bug: https://bugzilla.redhat.com/show_bug.cgi?id=1722330 about no ds-identify exists. ds-identify is running to check if any datasources can be found during boot, if no datasources available, it will disable cloud-init. I suggest Red Hat Linux add this ds-identify, then it should fix the bug https://bugzilla.redhat.com/show_bug.cgi?id=1593010
_Launchpad user **Eduardo Otubo(otubo)** wrote on 2019-08-20T06:31:48.822602+00:00_

@pengpengs, the first problem is indeed include ds-identify inside the rpm, this was the beginning of the solution. But then I included, and I found out that after the first boot (the second boot, and so on) cloud-init doesn't disable itself even if vmware datasource is not found. Which I believe is part of this issue - instead it treats itself as a new instance (iid reset) and resets the network configuration to default (dhcp)
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-08-20T08:16:06.649433+00:00_

@otubo, for this issue, cloud-init will be always enabled since ds-identify can find Open-VM-Tools installed and "disable_vmware_customization: false" from /etc/cloud/cloud.cfg file, with both of them, ds-identify will return OVF datasource and enable cloud-init. see code at https://github.com/cloud-init/cloud-init/blob/master/tools/ds-identify#L705

If you have included ds-identify to rpm, could you please check the generated /var/run/cloud-init/ds-identify.log on the Linux which cloud-init is enabled at second boot. Did it find another datasource?






_Launchpad user **Eduardo Otubo(otubo)** wrote on 2019-08-21T12:16:52.029090+00:00_

So what you're saying is that *before* my second boot I should set 'disable_vmware_customization: true', otherwise cloud-init will always be enable despite this bug?

Does it make sense to to have the function ovf_vmware_guest_customization() check for: Open-VM-Tools installed, 'disable_vmware_customization: false', if datasource is available *and* if the iid is still the same?  Adding these last two check would avoid me to change the configuration from true to false and form false to true everytime I need a customization. I would just need to attach the datasource and we would be good to go.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-08-26T03:55:06.882925+00:00_

If you have set 'disable_vmware_customization: false' in /etc/cloud/cloud.cfg, and had Open-VM-Tools installed and it's a Vitual Machine running on vSphere, you will hit this bug. 
So there are 3 conditions, remove anyone of them, cloud-init will NOT use DatasourceOVF to do customization.
ovf_vmware_guest_customization() is to check these 3 conditions and returns DataSourceOVF found when all 3 conditions are true.

I have a question on 'attach the datasource', are you attaching an ISO to CD-ROM?
_Launchpad user **Robbert Muller(mjrider)** wrote on 2019-09-04T14:10:10.537417+00:00_

Thinking a bit outside the box maybe

would getting the instance-id from the machine, just like vmware-guestinfo datasource, a better idea?
 
https://github.com/vmware/cloud-init-vmware-guestinfo/blob/99442e7ab784561c116caae54fe3a0c4a947a207/DataSourceVMwareGuestInfo.py#L160-L167

fudging around with a random instance id to recustomize a machine sounds like the wrong way around.


_Launchpad user **Eduardo Otubo(otubo)** wrote on 2019-09-19T08:14:01.659929+00:00_

@pengpengs I'm using the data source from vCenter "Policies and profile", there's no ISO or CDROM. Would you need a hand to help debugging this issue? I have access to vSphere cluster to test and create vms, etc.

Thanks.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-09-20T10:44:50.728061+00:00_

@Eduardo,
Yes, please help to debug this. I will also work on this soon.
The user data comes from /var/run/vmware-imc/cust.cfg not ISO or CDROM.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2019-11-08T06:06:23.839876+00:00_

VMware internal PR #2377600 is tracing this issue.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2020-06-15T13:13:03.123937+00:00_

@Ryan As you know, PR #229 (https://github.com/canonical/cloud-init/pull/229) author has not responded for months. To fix the issue, could you please help to implement the similar change on LOCAL datasource only?
_Launchpad user **Ryan Harper(raharper)** wrote on 2020-06-15T15:31:32.814152+00:00_

@Pengpeng

Have you tried testing this bug with PR #229 applied?

I suspect that it might just work since DataSourceOVF does not persist an obj.pkl

Per comment:

https://github.com/canonical/cloud-init/pull/229/commits/82d039b0496902ceef4e73586e533295903e2e31#r397500586

The issue there was that if the datasource persisted an object; then init-net stage would
use the existing object on disk and not use the fallback path.

For OVF, at local time, it will run _get_data over the datasource list, which after reboot per this bug, it not return True;  so I believe then that the fallback path should trigger as expected.

For Ec2 or other datasource which  do persist objects it's not clear what a fallback scenario looks like.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2020-06-16T10:02:40.434601+00:00_

@Ryan

Yes, I have verified the fallback path worked after PR #229 applied. so it should fix this issue.

I do NOT get "...DataSourceOVF does not persist an obj.pkl". From what I saw, after loaded DataSourceOVF, obj.pkl file was created under instance path. Doesn't the fallback ds load from obj.pkl?
_Launchpad user **Ryan Harper(raharper)** wrote on 2020-06-16T15:43:19.381517+00:00_

@Pengpeng,

> Yes, I have verified the fallback path worked after PR #229 applied. so it
> should fix this issue.

OK.  That's good news.   I think for local datasources, the fallback option
works quite well.

>
> I do NOT get "...DataSourceOVF does not persist an obj.pkl". From what I
> saw, after loaded DataSourceOVF, obj.pkl file was created under instance
> path. Doesn't the fallback ds load from obj.pkl?

Sorry.  I should have been more clear.  While the obj.pkl is written for
local datasources, it's only used in transitioning between local and net
stages:

For all datasources, cloud-init boot sequence does the following:

1) cloud-init init --local will purge the object cache unless it's told
   not too (manual cache clean); this removes the finished boot flag.
   and sets the 'existing' variable to 'check'; this is important for
   the next stage of cloud-init (cloud-init init)

   cloudinit/cmd/main.py:main_init lines 305 -> 317

   Now, cloud-init init --local will fetch the datasource with
   existing='check'

   cloudinit/stages.py:fetch() lines 349
     cloudinit/stages.py:_get_data_sources(existing='check') lines 236
       cloudinit/stages.py:_restore_from_checked_cache(existing='check') lines 211
         cloudinit/stages.py:_restore_from_cache() lines 184
           cloudinit/stages.py:_pkl_load() lines 943

    On reboot scenarios; the datasource is reloaded and then cloud-init
    compares the instance-id from /var/lib/cloud/data/instance_id to the
    ds.get_instance_id() value; if they match, then it's restored from cache

    If the do not match, then the datasource from cache (obj.pkl) is
    ignored.  Then cloud-init will walk the local datasources calling
    ._get_data() on each until a local datasource is found.

    Once a datasource is found (one of the datasources _get_data() method
    returns True, then self.datasource is set and we do persist an obj.pkl

    cloud-init init --local exits.


2) cloud-init runs after networking is up, this time in main.py:main_init
   ds mode is in NET mode which sets existing='trust';  the idea here
   is that cloud-init local mode detects and finds the correct datasource
   and so at NET stage we don't need to look for a datasource again if one
   was found.

   For *network only* datasources (Ec2 for example); ones that can only be
   detected by checking a network endpoint (for example
   http://169.254.169.254) will not be checked until networking is up; so in
   (1) we don't bother checking until this stage.

   When cloud-init attempts to restore_from_checked_cache() with
   existing='trust', then the pkl_load() succeeds.

   Next, cloud-init looks up the on-disk instance-id:
      /var/lib/cloud/data/instance-id
   And then calls ds.get_instance_id() and if they match, then we're on
   the same instance as we were before.  For local datasources; this is
   trivially true as when we found the datasource in (1) we persisted the
   object and we've just loaded it.


For local-only datasources (NoCloud, OVF); I think 229 works;  Specifically
for OVF since it's detection is based on files that are removed after first
boot.  I'd like to add a unittest to exercise this path, specifically we'd
want.

  a) /var/lib/cloud/* populated as it would look after a first boot
  b) run init_main with args.local=True
  c) mock datasource._get_data to return False (forcing down the fallback
  d) we should verify that self.datasource matches what's in obj.pkl


We should also test paths around OVF instance_id changing/resetting.  #229
operates under the assumption that the instance_id should not change between
reboots. A second unitest which ensure that if OVF instance_id changes that
the fallback path does NOT successful load OVF via fallback.


For Ec2, which is detected locally (by checking system UUID string), I don't
think we ever use the fallback path due to the local detection; this means
we always re-use the on-disk obj.pkl.   I don't think that's a blocker to
merging.


_Launchpad user **MikeN(mike-normi)** wrote on 2020-06-17T05:03:22.237307+00:00_

Just wanted to chime in here as a Vmware user that is suffering from the current implementation, to give some food for thought.

I do not really see what any of the proposed changes would fix. The root of all the issues is:

- Vmware customization using a file (vmware-imc) only supports network config, but no user-data or anything else, and is thus quite useless for cloud-init.
- Vmware customization using the OVF file supports user-data and is persistent, but does not support any network config, and is thus quite useless for setting up a VM as well.

What I do not understand, is why all this effort is not put in providing network config through the OVF file (which can be easily done by just adding a 'meta-data' base64 string next to the 'user-data' base64 string). A bug is open for this (#1247055) but there does not seem to be happening a lot there.

As the vmware-imc customization will never offer any user-data _and_ has the problem that is disappears, using that in any sensible way requires fixing the data source caching, but also fixing a merge between OVF data and IMC data (which is bug #1806133). If any of these issues are not fixed, it will only be possible to provide network config through IMC, but supplying any OVF user-data will always trigger a re-init on next reboot, breaking the system.

From the Vmware point of view, I do not understand why they don't put their efforts in getting cloud-init-vmware-guestinfo merged into the main cloud-init tree, and subsequently adding support for this in VCloud Director (which is unable to set these guestinfo properties at the moment).

I would be happy to look into supporting meta-data through the OVF file, and is seems quite trivial, but let me know if I'm missing something here and if it doesn't make sense.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2020-06-17T14:56:44.179799+00:00_

@Ryan,

Thanks a lot! You explained the stages clearly.
While I think you already know this: When I apply #229 in my test, I set fallback to True only when mode == sources.DSMODE_LOCAL.

-        init.fetch(existing=existing)
+        init.fetch(existing=existing,
+                   fallback=(mode == sources.DSMODE_LOCAL))

_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2020-06-17T15:12:06.947560+00:00_

@MikeN, I fully understand what you are suffering from. You are right about OVF can apply only meta-data from vmware-imc configuration or only user-data from OVF file. One solution for this could be vmware-imc customization offers both meta-data and user-data to cloud-init. 
_Launchpad user **Steffen Becker(snowball77)** wrote on 2020-09-22T18:22:09.144708+00:00_

I am facing the same issue. I have installed the latest focal cloud image using the user-data field to customize the VM and also set disable_vmware_customization: false for the first boot. I converted it into a template for cloning. It customizes correctly on first boot of a clone from the template. On reboot it falls back. 

In this setup it does not even matter that you remove disable_vmware_customization: again. Also with the removed config it does not persist the custom static IP on the reboot. Only creating /etc/cloud/cloud-init.disabled works and helps to persist the custom IP settings. However, then I cannot customize the VM a second time.

On OVF Details page, transfering the OVF via ISO is enabled in this setup, coming from the imported ova cloud image. If you want me to provide any logs I am happy to do so.
_Launchpad user **Pengpeng Sun(pengpengs)** wrote on 2020-09-23T07:27:14.987230+00:00_

Hi @Steffen,

Yes, this issue has not been fixed on the current latest cloud-init version 20.3, please refer VMware KB: https://kb.vmware.com/s/article/71264 to workaround this issue.

@Ryan, do you think this issue can be fixed in next cloud-init release.
_Launchpad user **Ben(bwatson1979)** wrote on 2020-10-02T11:09:28.701431+00:00_

Our process is to download the raw cloud-init image and upload it to a datastore in vSphere/vCenter as a template.  We then clone the template to a VM, create an ISO CD-ROM image with meta-data and user-data at the root of the ISO, attach the ISO to the CD-ROM of the VM, then power on the VM.

This process has served us well with Ubuntu 16.04 through 20.04 and for EL 7 (RHEL/CentOS).  It isn't until EL 8 that we've run into issues.  Various RedHat articles, issues, and bugzilla entries have landed me here.  I've recently tried RHEL 8.2 and still cannot get it to cloud-init with our process that has worked so well.

If you look at the running VM in the VMWare console, the screen immediately says "Probing EDD (edd off to disable)... ok" and sits like that indefinitely.  I'm not sure if my issues are related to this or not, but it is difficult to troubleshoot since I cannot even get a command prompt to get into the VM and see what's going on.
_Launchpad user **Eduardo Otubo(otubo)** wrote on 2021-04-13T12:01:08.763001+00:00_

Anyone have an update on this issue?
We have some RHEL instances on Power that are being affected.

Thanks!

Resolving patch:
diff --git a/cloudinit/sources/DataSourceVMware.py b/cloudinit/sources/DataSourceVMware.py
--- a/cloudinit/sources/DataSourceVMware.py
+++ b/cloudinit/sources/DataSourceVMware.py
@@ -176,7 +176,7 @@ def _get_data(self):
                 break
 
         if not self.data_access_method:
-            LOG.error("failed to find a valid data access method")
+            LOG.debug("failed to find a valid data access method")
             return False
 
         LOG.info("using data access method %s", self._get_subplatform())
@@ -270,6 +270,18 @@ def get_instance_id(self):
             self.metadata["instance-id"] = str(id_file.read()).rstrip().lower()
             return self.metadata["instance-id"]
 
+    def check_if_fallback_is_allowed(self):
+        if (
+            self.data_access_method
+            and self.data_access_method == DATA_ACCESS_METHOD_IMC
+            and is_vmware_platform()
+        ):
+            LOG.debug(
+                "Cache fallback is allowed for : %s", self._get_subplatform()
+            )
+            return True
+        return False
+
     def get_public_ssh_keys(self):
         for key_name in (
             "public-keys-data",
diff --git a/cloudinit/sources/__init__.py b/cloudinit/sources/__init__.py
--- a/cloudinit/sources/__init__.py
+++ b/cloudinit/sources/__init__.py
@@ -347,6 +347,9 @@ def _unpickle(self, ci_pkl_version: int) -> None:
             if not hasattr(self, key):
                 setattr(self, key, value)
 
+        if not hasattr(self, "check_if_fallback_is_allowed"):
+            setattr(self, "check_if_fallback_is_allowed", lambda: False)
+
         if hasattr(self, "userdata") and self.userdata is not None:
             # If userdata stores MIME data, on < python3.6 it will be
             # missing the 'policy' attribute that exists on >=python3.6.
@@ -926,6 +929,16 @@ def check_instance_id(self, sys_cfg):
         # quickly (local check only) if self.instance_id is still
         return False
 
+    def check_if_fallback_is_allowed(self):
+        """check_if_fallback_is_allowed()
+        Checks if a cached ds is allowed to be restored when no valid ds is
+        found in local mode by checking instance-id and searching valid data
+        through ds list.
+
+        @return True if a ds allows fallback, False otherwise.
+        """
+        return False
+
     @staticmethod
     def _determine_dsmode(candidates, default=None, valid=None):
         # return the first candidate that is non None, warn if not valid
diff --git a/cloudinit/stages.py b/cloudinit/stages.py
--- a/cloudinit/stages.py
+++ b/cloudinit/stages.py
@@ -359,20 +359,32 @@ def _get_data_source(self, existing) -> sources.DataSource:
             LOG.debug(myrep.description)
 
         if not ds:
-            util.del_file(self.paths.instance_link)
-            (cfg_list, pkg_list) = self._get_datasources()
-            # Deep copy so that user-data handlers can not modify
-            # (which will affect user-data handlers down the line...)
-            (ds, dsname) = sources.find_source(
-                self.cfg,
-                self.distro,
-                self.paths,
-                copy.deepcopy(self.ds_deps),
-                cfg_list,
-                pkg_list,
-                self.reporter,
-            )
-            LOG.info("Loaded datasource %s - %s", dsname, ds)
+            try:
+                cfg_list, pkg_list = self._get_datasources()
+                # Deep copy so that user-data handlers can not modify
+                # (which will affect user-data handlers down the line...)
+                ds, dsname = sources.find_source(
+                    self.cfg,
+                    self.distro,
+                    self.paths,
+                    copy.deepcopy(self.ds_deps),
+                    cfg_list,
+                    pkg_list,
+                    self.reporter,
+                )
+                util.del_file(self.paths.instance_link)
+                LOG.info("Loaded datasource %s - %s", dsname, ds)
+            except sources.DataSourceNotFoundException as e:
+                if existing != "check":
+                    raise e
+                ds = self._restore_from_cache()
+                if ds and ds.check_if_fallback_is_allowed():
+                    LOG.info(
+                        "Restored fallback datasource from checked cache: %s",
+                        ds,
+                    )
+                else:
+                    raise e
         self.datasource = ds
         # Ensure we adjust our path members datasource
         # now that we have one (thus allowing ipath to be used)


Test patch: