Repository: home-assistant/core (Python)

Issue:
mqtt_json device tracker broken - seems to try to start before MQTT integration has started and fails
### The problem

I use the mqtt based device tracker. It has worked stable and fine until 2023.4 when it fails to load

The error message says that MQTT is not enabled. Well. I have 100s of items based on MQTT in home assistant and they all work just fine

The error happens at restart of HA and the integration does not even attempt to try again. It just fails



### What version of Home Assistant Core has the issue?

core-2023.04.01

### What was the last working version of Home Assistant Core?

core-2023.03

### What type of installation are you running?

Home Assistant OS

### Integration causing the issue

mqtt_json

### Link to integration documentation on our website

https://www.home-assistant.io/integrations/mqtt_json/

### Diagnostics information

This is the error log

```
2023-04-08 20:24:49.158 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform legacy mqtt_json
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/device_tracker/legacy.py", line 288, in async_setup_legacy
setup = await self.platform.async_setup_scanner(
File "/usr/src/homeassistant/homeassistant/components/mqtt_json/device_tracker.py", line 76, in async_setup_scanner
await mqtt.async_subscribe(hass, topic, async_message_received, qos)
File "/usr/src/homeassistant/homeassistant/components/mqtt/client.py", line 166, in async_subscribe
raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Cannot subscribe to topic 'location/testtracker', MQTT is not enabled
```

### Example YAML snippet

```yaml
This is the config in yaml

device_tracker:
  - platform: mqtt_json
    devices:
      testtracker: 'location/testtracker'
```


### Anything in the logs that might be useful for us?

_No response_

### Additional information

_No response_


Discussion: [mqtt_json documentation](https://www.home-assistant.io/integrations/mqtt_json)
[mqtt_json source](https://github.com/home-assistant/core/tree/dev/homeassistant/components/mqtt_json)
I am seeing the same thing, and it is inconsistent.  It sometimes takes 2 or 3 restarts, but it eventually works without errors. 
I have not seen it succeed even once. It probably depends on how large an MQTT setup you have. It seems the code should be waiting for mqtt instead of failing. Maybe it waits but then it gives up too quickly
I asked in the Discored developer general channel for help assigning a developer as it seems the manifast has none defined

I can add one more data point. This failing integration reports a start up time of 0:00 so it fails immediately. I think it tried to query the MQTT server before the rest of the system is up and fails. There is some runtime dependency that seems lost
@jbouwh 
@emontnemery 

I just raised a bug for an other MQTT related bug and you got auto assigned. Isn't this one supposed to be assigned to you? If not then I am sorry
I'll have a look a this one.
I ran the boot with all of HA in debug logging

I cannot see anything obvious but again context may help.

```
2023-04-10 12:48:38.831 DEBUG (MainThread) [synology_dsm.synology_dsm] Request Method: GET
2023-04-10 12:48:38.833 DEBUG (MainThread) [homeassistant.bootstrap] Running timeout Zones: {'sensor': <sensor: 2 / 0>, 'frigate': <frigate: 0 / 1>, 'tasmota': <tasmota: 0 / 1>, 'rtsp_to_webrtc': <rtsp_to_webrtc: 0 / 1>, 'switch': <switch: 0 / 1>, 'mobile_app': <mobile_app: 0 / 1>, 'default_config': <default_config: 0 / 1>, 'energidataservice': <energidataservice: 0 / 1>, 'radio_browser': <radio_browser: 0 / 1>, 'stream': <stream: 0 / 1>, 'squeezebox': <squeezebox: 0 / 1>, 'notify': <notify: 1 / 0>, 'ios': <ios: 0 / 1>, 'template': <template: 0 / 3>, 'watchman': <watchman: 0 / 1>, 'broadlink': <broadlink: 0 / 1>, 'discovery': <discovery: 0 / 1>, 'hacs': <hacs: 0 / 1>, 'camera': <camera: 0 / 1>, 'generic': <generic: 0 / 1>, 'heos': <heos: 0 / 1>, 'utility_meter': <utility_meter: 0 / 1>, 'deconz': <deconz: 0 / 1>, 'zwave_js': <zwave_js: 0 / 1>, 'esphome': <esphome: 0 / 1>, 'cast': <cast: 0 / 1>, 'kodi': <kodi: 0 / 1>, 'miele': <miele: 0 / 1>, 'google': <google: 0 / 1>, 'onvif': <onvif: 0 / 1>, 'device_tracker': <device_tracker: 1 / 0>, 'binary_sensor': <binary_sensor: 2 / 0>}
2023-04-10 12:48:38.834 DEBUG (MainThread) [homeassistant.bootstrap] Integration remaining: {'sensor.nordpool': 9.76872, 'group': 9.734051, 'switch_as_x': 9.725189, 'sun': 9.530145, 'notify': 9.499429, 'samsungtv': 3.560575, 'synology_dsm': 3.288519, 'brother': 2.849421, 'dlna_dms': 2.469132, 'denonavr': 1.569922, 'ipp': 1.340537, 'unifi': 1.162033, 'alexa_media': 0.794265, 'mqtt': 0.703801, 'meater': 0.677947, 'device_tracker': 0.618813, 'binary_sensor.group': 0.569354, 'octoprint': 0.058862, 'sensor.sun': 0.035544}
2023-04-10 12:48:38.837 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=binary_sensor.doors_windows, old_state=None, new_state=<state binary_sensor.doors_windows=unavailable; friendly_name=Doors & Windows @ 2023-04-10T12:48:38.837509+02:00>>
2023-04-10 12:48:38.838 INFO (MainThread) [homeassistant.components.device_tracker] Setting up device_tracker.mqtt_json
2023-04-10 12:48:38.839 DEBUG (SyncWorker_1) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/logbook/translations/en.json
2023-04-10 12:48:38.839 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/logbook/translations/en-GB.json
2023-04-10 12:48:38.840 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/input_datetime/translations/en-GB.json
2023-04-10 12:48:38.840 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/system_health/translations/en-GB.json
2023-04-10 12:48:38.841 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/input_text/translations/en-GB.json
2023-04-10 12:48:38.841 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/schedule/translations/en-GB.json
2023-04-10 12:48:38.841 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/input_button/translations/en-GB.json
2023-04-10 12:48:38.842 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/input_select/translations/en-GB.json
2023-04-10 12:48:38.843 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/uptime/translations/sensor.en-GB.json
2023-04-10 12:48:38.843 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/synology_dsm/translations/en-GB.json
2023-04-10 12:48:38.843 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/media_source/translations/en-GB.json
2023-04-10 12:48:38.843 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/brother/translations/en-GB.json
2023-04-10 12:48:38.844 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/light/translations/en-GB.json
2023-04-10 12:48:38.844 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/scene/translations/en-GB.json
2023-04-10 12:48:38.845 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/conversation/translations/en-GB.json
2023-04-10 12:48:38.845 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/dlna_dms/translations/en-GB.json
2023-04-10 12:48:38.846 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/timer/translations/en-GB.json
2023-04-10 12:48:38.847 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/input_number/translations/en-GB.json
2023-04-10 12:48:38.847 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/counter/translations/en-GB.json
2023-04-10 12:48:38.847 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/ffmpeg/translations/en-GB.json
2023-04-10 12:48:38.838 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform legacy mqtt_json
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/device_tracker/legacy.py", line 288, in async_setup_legacy
    setup = await self.platform.async_setup_scanner(
  File "/usr/src/homeassistant/homeassistant/components/mqtt_json/device_tracker.py", line 76, in async_setup_scanner
    await mqtt.async_subscribe(hass, topic, async_message_received, qos)
  File "/usr/src/homeassistant/homeassistant/components/mqtt/client.py", line 166, in async_subscribe
    raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Cannot subscribe to topic 'location/testtracker', MQTT is not enabled
2023-04-10 12:48:38.848 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/denonavr/translations/en-GB.json
2023-04-10 12:48:38.849 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/input_boolean/translations/en-GB.json
2023-04-10 12:48:38.849 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/wake_on_lan/translations/en-GB.json
2023-04-10 12:48:38.849 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/application_credentials/translations/en-GB.json
2023-04-10 12:48:38.850 DEBUG (SyncWorker_6) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/zone/translations/en-GB.json
2023-04-10 12:48:38.852 DEBUG (SyncWorker_1) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/uptime/translations/sensor.en.json
2023-04-10 12:48:38.855 DEBUG (SyncWorker_1) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/media_source/translations/en.json
2023-04-10 12:48:38.865 DEBUG (SyncWorker_1) [homeassistant.util.json] JSON file not found: /usr/src/homeassistant/homeassistant/components/ffmpeg/translations/en.json
2023-04-10 12:48:38.882 INFO (MainThread) [homeassistant.setup] Setting up energidataservice
```



Fix (patch):
diff --git a/homeassistant/components/manual_mqtt/alarm_control_panel.py b/homeassistant/components/manual_mqtt/alarm_control_panel.py
index fd6adb009aaefc..adb251bd71a6e0 100644
--- a/homeassistant/components/manual_mqtt/alarm_control_panel.py
+++ b/homeassistant/components/manual_mqtt/alarm_control_panel.py
@@ -187,13 +187,19 @@ def _state_schema(state):
 )
 
 
-def setup_platform(
+async def async_setup_platform(
     hass: HomeAssistant,
     config: ConfigType,
     add_entities: AddEntitiesCallback,
     discovery_info: DiscoveryInfoType | None = None,
 ) -> None:
     """Set up the manual MQTT alarm platform."""
+    # Make sure MQTT integration is enabled and the client is available
+    # We cannot count on dependencies as the alarm_control_panel platform setup
+    # also will be triggered when mqtt is loading the `alarm_control_panel` platform
+    if not await mqtt.async_wait_for_mqtt_client(hass):
+        _LOGGER.error("MQTT integration is not available")
+        return
     add_entities(
         [
             ManualMQTTAlarm(
diff --git a/homeassistant/components/mqtt/__init__.py b/homeassistant/components/mqtt/__init__.py
index e588acec53a813..582608335593fc 100644
--- a/homeassistant/components/mqtt/__init__.py
+++ b/homeassistant/components/mqtt/__init__.py
@@ -68,6 +68,7 @@
     CONF_WS_HEADERS,
     CONF_WS_PATH,
     DATA_MQTT,
+    DATA_MQTT_AVAILABLE,
     DEFAULT_DISCOVERY,
     DEFAULT_ENCODING,
     DEFAULT_PREFIX,
@@ -87,8 +88,9 @@
     ReceiveMessage,
     ReceivePayloadType,
 )
-from .util import (
+from .util import (  # noqa: F401
     async_create_certificate_temp_files,
+    async_wait_for_mqtt_client,
     get_mqtt_data,
     mqtt_config_entry_enabled,
     valid_publish_topic,
@@ -183,34 +185,54 @@ async def _async_config_entry_updated(hass: HomeAssistant, entry: ConfigEntry) -
 
 async def async_setup_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool:
     """Load a config entry."""
-    conf = dict(entry.data)
-    # Fetch configuration
-    hass_config = await conf_util.async_hass_config_yaml(hass)
-    mqtt_yaml = PLATFORM_CONFIG_SCHEMA_BASE(hass_config.get(DOMAIN, {}))
-    client = MQTT(hass, entry, conf)
-    if DOMAIN in hass.data:
-        mqtt_data = get_mqtt_data(hass)
-        mqtt_data.config = mqtt_yaml
-        mqtt_data.client = client
-    else:
-        # Initial setup
-        websocket_api.async_register_command(hass, websocket_subscribe)
-        websocket_api.async_register_command(hass, websocket_mqtt_info)
-        hass.data[DATA_MQTT] = mqtt_data = MqttData(config=mqtt_yaml, client=client)
-    client.start(mqtt_data)
-
-    await async_create_certificate_temp_files(hass, dict(entry.data))
-    # Restore saved subscriptions
-    if mqtt_data.subscriptions_to_restore:
-        mqtt_data.client.async_restore_tracked_subscriptions(
-            mqtt_data.subscriptions_to_restore
+    conf: dict[str, Any]
+    mqtt_data: MqttData
+
+    async def _setup_client() -> tuple[MqttData, dict[str, Any]]:
+        """Set up the MQTT client."""
+        # Fetch configuration
+        conf = dict(entry.data)
+        hass_config = await conf_util.async_hass_config_yaml(hass)
+        mqtt_yaml = PLATFORM_CONFIG_SCHEMA_BASE(hass_config.get(DOMAIN, {}))
+        client = MQTT(hass, entry, conf)
+        if DOMAIN in hass.data:
+            mqtt_data = get_mqtt_data(hass)
+            mqtt_data.config = mqtt_yaml
+            mqtt_data.client = client
+        else:
+            # Initial setup
+            websocket_api.async_register_command(hass, websocket_subscribe)
+            websocket_api.async_register_command(hass, websocket_mqtt_info)
+            hass.data[DATA_MQTT] = mqtt_data = MqttData(config=mqtt_yaml, client=client)
+        client.start(mqtt_data)
+
+        await async_create_certificate_temp_files(hass, dict(entry.data))
+        # Restore saved subscriptions
+        if mqtt_data.subscriptions_to_restore:
+            mqtt_data.client.async_restore_tracked_subscriptions(
+                mqtt_data.subscriptions_to_restore
+            )
+            mqtt_data.subscriptions_to_restore = []
+        mqtt_data.reload_dispatchers.append(
+            entry.add_update_listener(_async_config_entry_updated)
         )
-        mqtt_data.subscriptions_to_restore = []
-    mqtt_data.reload_dispatchers.append(
-        entry.add_update_listener(_async_config_entry_updated)
-    )
 
-    await mqtt_data.client.async_connect()
+        await mqtt_data.client.async_connect()
+        return (mqtt_data, conf)
+
+    client_available: asyncio.Future[bool]
+    if DATA_MQTT_AVAILABLE not in hass.data:
+        client_available = hass.data[DATA_MQTT_AVAILABLE] = asyncio.Future()
+    else:
+        client_available = hass.data[DATA_MQTT_AVAILABLE]
+
+    setup_ok: bool = False
+    try:
+        mqtt_data, conf = await _setup_client()
+        setup_ok = True
+    finally:
+        if not client_available.done():
+            client_available.set_result(setup_ok)
 
     async def async_publish_service(call: ServiceCall) -> None:
         """Handle MQTT publish service calls."""
@@ -565,6 +587,9 @@ async def async_unload_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool:
         registry_hooks.popitem()[1]()
     # Wait for all ACKs and stop the loop
     await mqtt_client.async_disconnect()
+
+    # Cleanup MQTT client availability
+    hass.data.pop(DATA_MQTT_AVAILABLE, None)
     # Store remaining subscriptions to be able to restore or reload them
     # when the entry is set up again
     if subscriptions := mqtt_client.subscriptions:
diff --git a/homeassistant/components/mqtt/const.py b/homeassistant/components/mqtt/const.py
index fe7c6eb4cf0979..c91c54a79a41d2 100644
--- a/homeassistant/components/mqtt/const.py
+++ b/homeassistant/components/mqtt/const.py
@@ -35,6 +35,7 @@
 CONF_TLS_INSECURE = "tls_insecure"
 
 DATA_MQTT = "mqtt"
+DATA_MQTT_AVAILABLE = "mqtt_client_available"
 
 DEFAULT_PREFIX = "homeassistant"
 DEFAULT_BIRTH_WILL_TOPIC = DEFAULT_PREFIX + "/status"
diff --git a/homeassistant/components/mqtt/util.py b/homeassistant/components/mqtt/util.py
index 155756a58dcba1..896ba21f8022ec 100644
--- a/homeassistant/components/mqtt/util.py
+++ b/homeassistant/components/mqtt/util.py
@@ -2,13 +2,16 @@
 
 from __future__ import annotations
 
+import asyncio
 import os
 from pathlib import Path
 import tempfile
 from typing import Any
 
+import async_timeout
 import voluptuous as vol
 
+from homeassistant.config_entries import ConfigEntryState
 from homeassistant.core import HomeAssistant
 from homeassistant.helpers import config_validation as cv, template
 from homeassistant.helpers.typing import ConfigType
@@ -22,6 +25,7 @@
     CONF_CLIENT_CERT,
     CONF_CLIENT_KEY,
     DATA_MQTT,
+    DATA_MQTT_AVAILABLE,
     DEFAULT_ENCODING,
     DEFAULT_QOS,
     DEFAULT_RETAIN,
@@ -29,6 +33,8 @@
 )
 from .models import MqttData
 
+AVAILABILITY_TIMEOUT = 30.0
+
 TEMP_DIR_NAME = f"home-assistant-{DOMAIN}"
 
 _VALID_QOS_SCHEMA = vol.All(vol.Coerce(int), vol.In([0, 1, 2]))
@@ -41,6 +47,37 @@ def mqtt_config_entry_enabled(hass: HomeAssistant) -> bool | None:
     return not bool(hass.config_entries.async_entries(DOMAIN)[0].disabled_by)
 
 
+async def async_wait_for_mqtt_client(hass: HomeAssistant) -> bool:
+    """Wait for the MQTT client to become available.
+
+    Waits when mqtt set up is in progress,
+    It is not needed that the client is connected.
+    Returns True if the mqtt client is available.
+    Returns False when the client is not available.
+    """
+    if not mqtt_config_entry_enabled(hass):
+        return False
+
+    entry = hass.config_entries.async_entries(DOMAIN)[0]
+    if entry.state == ConfigEntryState.LOADED:
+        return True
+
+    state_reached_future: asyncio.Future[bool]
+    if DATA_MQTT_AVAILABLE not in hass.data:
+        hass.data[DATA_MQTT_AVAILABLE] = state_reached_future = asyncio.Future()
+    else:
+        state_reached_future = hass.data[DATA_MQTT_AVAILABLE]
+        if state_reached_future.done():
+            return state_reached_future.result()
+
+    try:
+        async with async_timeout.timeout(AVAILABILITY_TIMEOUT):
+            # Await the client setup or an error state was received
+            return await state_reached_future
+    except asyncio.TimeoutError:
+        return False
+
+
 def valid_topic(topic: Any) -> str:
     """Validate that this is a valid topic name/filter."""
     validated_topic = cv.string(topic)
diff --git a/homeassistant/components/mqtt_json/device_tracker.py b/homeassistant/components/mqtt_json/device_tracker.py
index 2c67751551c99d..2b355eb68e6296 100644
--- a/homeassistant/components/mqtt_json/device_tracker.py
+++ b/homeassistant/components/mqtt_json/device_tracker.py
@@ -47,6 +47,13 @@ async def async_setup_scanner(
     discovery_info: DiscoveryInfoType | None = None,
 ) -> bool:
     """Set up the MQTT JSON tracker."""
+    # Make sure MQTT integration is enabled and the client is available
+    # We cannot count on dependencies as the device_tracker platform setup
+    # also will be triggered when mqtt is loading the `device_tracker` platform
+    if not await mqtt.async_wait_for_mqtt_client(hass):
+        _LOGGER.error("MQTT integration is not available")
+        return False
+
     devices = config[CONF_DEVICES]
     qos = config[CONF_QOS]
 
diff --git a/homeassistant/components/mqtt_room/sensor.py b/homeassistant/components/mqtt_room/sensor.py
index b1b52e42fce3b2..00441690b47f90 100644
--- a/homeassistant/components/mqtt_room/sensor.py
+++ b/homeassistant/components/mqtt_room/sensor.py
@@ -68,6 +68,12 @@ async def async_setup_platform(
     discovery_info: DiscoveryInfoType | None = None,
 ) -> None:
     """Set up MQTT room Sensor."""
+    # Make sure MQTT integration is enabled and the client is available
+    # We cannot count on dependencies as the sensor platform setup
+    # also will be triggered when mqtt is loading the `sensor` platform
+    if not await mqtt.async_wait_for_mqtt_client(hass):
+        _LOGGER.error("MQTT integration is not available")
+        return
     async_add_entities(
         [
             MQTTRoomSensor(
diff --git a/homeassistant/components/snips/__init__.py b/homeassistant/components/snips/__init__.py
index d4619fa3b3a33e..ac26a5bc8dc01b 100644
--- a/homeassistant/components/snips/__init__.py
+++ b/homeassistant/components/snips/__init__.py
@@ -90,12 +90,8 @@
 
 async def async_setup(hass: HomeAssistant, config: ConfigType) -> bool:
     """Activate Snips component."""
-    # Make sure MQTT is available and the entry is loaded
-    if not hass.config_entries.async_entries(
-        mqtt.DOMAIN
-    ) or not await hass.config_entries.async_wait_component(
-        hass.config_entries.async_entries(mqtt.DOMAIN)[0]
-    ):
+    # Make sure MQTT integration is enabled and the client is available
+    if not await mqtt.async_wait_for_mqtt_client(hass):
         _LOGGER.error("MQTT integration is not available")
         return False