Repository: home-assistant/core
Pull request: Create MQTT device referenced by via device
<!--
  You are amazing! Thanks for contributing to our project!
  Please, DO NOT DELETE ANY TEXT from this template! (unless instructed).
-->
## Proposed change
<!--
  Describe the big picture of your changes here to communicate to the
  maintainers why we should accept this pull request. If it fixes a bug
  or resolves a feature request, be sure to link to that issue in the
  additional information section.
-->
MQTT discovery order is not deterministic, and a device referenced as `via_device` may be discovered after devices referencing it.

This PR creates the referenced device if it does not exist.

The MQTT item can be either an MQTT entity, MQTT tag or MQTT device trigger.

Tests are added to see a via device is available if it is referenced after initial discovery or, after a discovery update that sets the via device.

## Type of change
<!--
  What type of change does your PR introduce to Home Assistant?
  NOTE: Please, check only 1! box!
  If your PR requires multiple boxes to be checked, you'll most likely need to
  split it into multiple PRs. This makes things easier and faster to code review.
-->

- [ ] Dependency upgrade
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New integration (thank you!)
- [ ] New feature (which adds functionality to an existing integration)
- [ ] Deprecation (breaking change to happen in the future)
- [ ] Breaking change (fix/feature causing existing functionality to break)
- [ ] Code quality improvements to existing code or addition of tests

## Additional information
<!--
  Details are important, and help maintainers processing your PR.
  Please be sure to fill out additional details, if applicable.
-->

- This PR fixes or closes issue: fixes #131551
- This PR is related to issue: 
- Link to documentation pull request: 

## Checklist
<!--
  Put an `x` in the boxes that apply. You can also fill these out after
  creating the PR. If you're unsure about any of them, don't hesitate to ask.
  We're here to help! This is simply a reminder of what we are going to look
  for before merging your code.
-->

- [x] The code change is tested and works locally.
- [x] Local tests pass. **Your PR cannot be merged unless tests pass**
- [x] There is no commented out code in this PR.
- [x] I have followed the [development checklist][dev-checklist]
- [x] I have followed the [perfect PR recommendations][perfect-pr]
- [x] The code has been formatted using Ruff (`ruff format homeassistant tests`)
- [x] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

- [ ] Documentation added/updated for [www.home-assistant.io][docs-repository]

If the code communicates with devices, web services, or third-party tools:

- [ ] The [manifest file][manifest-docs] has all fields filled out correctly.  
      Updated and included derived files by running: `python3 -m script.hassfest`.
- [ ] New or updated dependencies have been added to `requirements_all.txt`.  
      Updated by running `python3 -m script.gen_requirements_all`.
- [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

<!--
  This project is very active and we have a high turnover of pull requests.

  Unfortunately, the number of incoming pull requests is higher than what our
  reviewers can review and merge so there is a long backlog of pull requests
  waiting for review. You can help here!
  
  By reviewing another pull request, you will help raise the code quality of
  that pull request and the final review will be faster. This way the general
  pace of pull request reviews will go up and your wait time will go down.
  
  When picking a pull request to review, try to choose one that hasn't yet
  been reviewed.

  Thanks for helping out!
-->

To help with the load of incoming pull requests:

- [ ] I have reviewed two other [open pull requests][prs] in this repository.

[prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure

<!--
  Thank you for contributing <3

  Below, some useful links you could explore:
-->
[dev-checklist]: https://developers.home-assistant.io/docs/development_checklist/
[manifest-docs]: https://developers.home-assistant.io/docs/creating_integration_manifest/
[quality-scale]: https://developers.home-assistant.io/docs/integration_quality_scale_index/
[docs-repository]: https://github.com/home-assistant/home-assistant.io
[perfect-pr]: https://developers.home-assistant.io/docs/review-process/#creating-the-perfect-pr


Problem statement:
MQTT discovery via_device not recognized 
### The problem

When sending a MQTT discovery message including via_device is sent, when the via_device is not available (yet), the device is not shown connected, after the via_device is sent at a later time.

Moreover, if the first discovery message is sent again, with the via_device available, it remains disconnected forever.

Steps to reproduce:

Send discovery message: **homeassistant/light/lamp1/config**
```
{
	"name": "Lamp 1",
	"object_id": "lamp1",
	"unique_id": "lamp1",
	"command_topic": "sometopic",
	"device": {
		"identifiers": "mydevice1",
		"name": "Device 1",
		"via_device": "topleveldevice1"
	}
}
```

Now send another message including the via_device: **homeassistant/sensor/sensor1/config**
```
{
	"name": "Sensor 1",
	"object_id": "sensor1",
	"unique_id": "sensor1",
	"state_topic": "othertopic",
	"device_class": "temperature",
	"unit_of_measurement": "°C",
	"device": {
		"identifiers": "topleveldevice1",
		"name": "Top level device 1",
		"model": "Model",
		"manufacturer": "Someone"
	}
}
```

The first Device 1 is not show connected to the via_device "Top level device 1".

When trying to fix this you can try to re-send the first message, but it will remain disconnected.

To compare: when you first send the message with the via_device, and later connected device, it is connected correctly.

So the order of discovery message is critical. This should not be the case, or it should be documented.


### What version of Home Assistant Core has the issue?

2024.7.3

### What was the last working version of Home Assistant Core?

_No response_

### What type of installation are you running?

Home Assistant Container

### Integration causing the issue

MQTT

### Link to integration documentation on our website

_No response_

### Diagnostics information

_No response_

### Example YAML snippet

_No response_

### Anything in the logs that might be useful for us?

_No response_

### Additional information

_No response_

Hints from issue discussion:
Hey there @emontnemery, @jbouwh, @bdraco, mind taking a look at this issue as it has been labeled with an integration (`mqtt`) you are listed as a [code owner](https://github.com/home-assistant/core/blob/dev/CODEOWNERS#L954) for? Thanks!

<details>
<summary>Code owner commands</summary>


Code owners of `mqtt` can trigger bot actions by commenting:

- `@home-assistant close` Closes the issue.
- `@home-assistant rename Awesome new title` Renames the issue.
- `@home-assistant reopen` Reopen the issue.
- `@home-assistant unassign mqtt` Removes the current integration label and assignees on the issue, add the integration domain after the command.
- `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
- `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

</details>


<sub><sup>(message by CodeOwnersMention)</sup></sub>

---

[mqtt documentation](https://www.home-assistant.io/integrations/mqtt)
[mqtt source](https://github.com/home-assistant/core/tree/dev/homeassistant/components/mqtt)
<sub><sup>(message by IssueLinks)</sup></sub>
Resending a discovery message will only update the entity if the message it self was changed. Aftter a reload all device releations should show fine because they are all added to the device registry.
Not sure what you mean by reload?

The order in which you send the messages has a PERMANENT effect on the device structure. 

Please try the provided messages, and you will see the problem.
Discovery message need to be resend after The Home Assistant mqtt integration is (re)started. This can be either triggered by:
- Subscribing to the birth message and use that as a trigger
- Retained discovery messages

https://www.home-assistant.io/integrations/mqtt/#discovery-messages-en-availability

MQTT will recognize the via device next time the entity is loaded.

If you want to fix this initially you either need to:
- submit the discovery message in the correct order, or
- update the discovery message, so that there is a delta and the entity and device registry is updated

We filter out discovery messages by design if they did not change.
I have fixed this for my own device (button.plus), all I want to do is help the stability of HA.

The discovery service feels quite unpredictable and unstable, partly due to the case at hand.

I invite you to try the sample I included, and you will understand.

If you filter discovery messages if the are not changed, this is a problem if the context is changed, e.g. when via_device is added.


I understand and respect your concern. The `via_device` option though only works if the `via_device` exists. As the device registry initially will be updated for all devices, it might be the `via_device` link could not be resolved. After the MQTT integration was restarted or reloaded, this dependency will be resolved, after processing the discovery messages for the second time.

> If you filter discovery messages if the are not changed, this is a problem if the context is changed, e.g. when via_device is added.

If we would not filter those out we would this could cause a perfomance impact if discovery messages are re-send. As this is a temporary issue, and solvable by sending the message in the correct order. Therefore I do not think we should change this logic.
Please Jan, if you respect my concern, take 2 minutes and try my example.

Just send the two messages, and restart the service: the devices remain disconnected!
> Please Jan, if you respect my concern, take 2 minutes and try my example.
> 
> Just send the two messages, and restart the service: the devices remain disconnected!

![image](https://github.com/user-attachments/assets/0fd8cbe5-121b-4d48-81dc-d7b11b90c7aa)

After reloading mqtt the device info is correctly shown.
Note that I used retained configuration payloads to replay the configurations
Please note that this is NOT about getting the devices connected. You can simply achieve the connection (as show in your picture) by sending the messages in reverse order (first Sensor 1 and then Lamp 1).

This issues is about PERMANENT disconnection of the devices, even after multiple reloading, and multiple resubmission.

When using retained messages, the order of replay is not guaranteed. I guess the replay was in the correct order?
BTW I'm using the homeassistant/status topic to replay the discovery messages. 

But auto discovery should really be independent of message order.  It makes you think you've gone mad sometimes, really. Until I found out about this ordering dependency, after hours of trying and testing.
> When using retained messages, the order of replay is not guaranteed. I guess the replay was in the correct order?
> BTW I'm using the homeassistant/status topic to replay the discovery messages.

The order does not matter when the payloads are replayed, because the devices are already in the device registry by then.

Commit message: Fix tests - do not mutate tests constants

Patch to review:
diff --git a/homeassistant/components/mqtt/entity.py b/homeassistant/components/mqtt/entity.py
index 46b2c9e1d42b5e..f7a49482627743 100644
--- a/homeassistant/components/mqtt/entity.py
+++ b/homeassistant/components/mqtt/entity.py
@@ -1185,6 +1185,24 @@ def device_info_from_specifications(
     return info
 
 
+@callback
+def ensure_via_device_exists(
+    hass: HomeAssistant, device_info: DeviceInfo | None, config_entry: ConfigEntry
+) -> None:
+    """Ensore the via device is in the device registry."""
+    if device_info is None:
+        return
+    device_registry = dr.async_get(hass)
+    if CONF_VIA_DEVICE in device_info and not device_registry.async_get_device(
+        identifiers={device_info["via_device"]}
+    ):
+        # Ensure the via device exists in the device registry
+        device_registry.async_get_or_create(
+            config_entry_id=config_entry.entry_id,
+            identifiers={device_info["via_device"]},
+        )
+
+
 class MqttEntityDeviceInfo(Entity):
     """Mixin used for mqtt platforms that support the device registry."""
 
@@ -1203,6 +1221,7 @@ def device_info_discovery_update(self, config: DiscoveryInfoType) -> None:
         device_info = self.device_info
 
         if device_info is not None:
+            ensure_via_device_exists(self.hass, device_info, self._config_entry)
             device_registry.async_get_or_create(
                 config_entry_id=config_entry_id, **device_info
             )
@@ -1256,6 +1275,7 @@ def __init__(
             self, hass, discovery_data, self.discovery_update
         )
         MqttEntityDeviceInfo.__init__(self, config.get(CONF_DEVICE), config_entry)
+        ensure_via_device_exists(self.hass, self.device_info, self._config_entry)
 
     def _init_entity_id(self) -> None:
         """Set entity_id from object_id if defined in config."""
@@ -1490,6 +1510,8 @@ def update_device(
     config_entry_id = config_entry.entry_id
     device_info = device_info_from_specifications(config[CONF_DEVICE])
 
+    ensure_via_device_exists(hass, device_info, config_entry)
+
     if config_entry_id is not None and device_info is not None:
         update_device_info = cast(dict[str, Any], device_info)
         update_device_info["config_entry_id"] = config_entry_id
diff --git a/tests/components/mqtt/test_discovery.py b/tests/components/mqtt/test_discovery.py
index e49e7a27c8d46a..8a674a4e1cdbfd 100644
--- a/tests/components/mqtt/test_discovery.py
+++ b/tests/components/mqtt/test_discovery.py
@@ -2987,3 +2987,139 @@ async def test_shared_state_topic(
     state = hass.states.get(entity_id)
     assert state is not None
     assert state.state == "New state3"
+
+
+@pytest.mark.parametrize("single_configs", [copy.deepcopy(TEST_SINGLE_CONFIGS)])
+async def test_discovery_with_late_via_device_discovery(
+    hass: HomeAssistant,
+    device_registry: dr.DeviceRegistry,
+    mqtt_mock_entry: MqttMockHAClientGenerator,
+    tag_mock: AsyncMock,
+    single_configs: list[tuple[str, dict[str, Any]]],
+) -> None:
+    """Test a via device is available and the discovery of the via device is late."""
+    await mqtt_mock_entry()
+
+    await hass.async_block_till_done()
+    await hass.async_block_till_done()
+
+    via_device_entry = device_registry.async_get_device(
+        {("mqtt", "id_via_very_unique")}
+    )
+    assert via_device_entry is None
+    # Discovery single config schema
+    for discovery_topic, config in single_configs:
+        config["device"]["via_device"] = "id_via_very_unique"
+        payload = json.dumps(config)
+        async_fire_mqtt_message(
+            hass,
+            discovery_topic,
+            payload,
+        )
+        via_device_entry = device_registry.async_get_device(
+            {("mqtt", "id_via_very_unique")}
+        )
+        assert via_device_entry is not None
+        assert via_device_entry.name is None
+
+    await hass.async_block_till_done()
+
+    # Now discover the via device (a switch)
+    via_device_config = {
+        "name": None,
+        "command_topic": "test-switch-topic",
+        "unique_id": "very_unique_switch",
+        "device": {"identifiers": ["id_via_very_unique"], "name": "My Switch"},
+    }
+    payload = json.dumps(via_device_config)
+    via_device_discovery_topic = "homeassistant/switch/very_unique/config"
+    async_fire_mqtt_message(
+        hass,
+        via_device_discovery_topic,
+        payload,
+    )
+    await hass.async_block_till_done()
+    await hass.async_block_till_done()
+    via_device_entry = device_registry.async_get_device(
+        {("mqtt", "id_via_very_unique")}
+    )
+    assert via_device_entry is not None
+    assert via_device_entry.name == "My Switch"
+
+    await help_check_discovered_items(hass, device_registry, tag_mock)
+
+
+@pytest.mark.parametrize("single_configs", [copy.deepcopy(TEST_SINGLE_CONFIGS)])
+async def test_discovery_with_late_via_device_update(
+    hass: HomeAssistant,
+    device_registry: dr.DeviceRegistry,
+    mqtt_mock_entry: MqttMockHAClientGenerator,
+    tag_mock: AsyncMock,
+    single_configs: list[tuple[str, dict[str, Any]]],
+) -> None:
+    """Test a via device is available and the discovery of the via device is is set via an update."""
+    await mqtt_mock_entry()
+
+    await hass.async_block_till_done()
+    await hass.async_block_till_done()
+
+    via_device_entry = device_registry.async_get_device(
+        {("mqtt", "id_via_very_unique")}
+    )
+    assert via_device_entry is None
+    # Discovery single config schema without via device
+    for discovery_topic, config in single_configs:
+        payload = json.dumps(config)
+        async_fire_mqtt_message(
+            hass,
+            discovery_topic,
+            payload,
+        )
+        via_device_entry = device_registry.async_get_device(
+            {("mqtt", "id_via_very_unique")}
+        )
+        await hass.async_block_till_done()
+        await hass.async_block_till_done()
+        assert via_device_entry is None
+
+    # Resend the discovery update to set the via device
+    for discovery_topic, config in single_configs:
+        config["device"]["via_device"] = "id_via_very_unique"
+        payload = json.dumps(config)
+        async_fire_mqtt_message(
+            hass,
+            discovery_topic,
+            payload,
+        )
+        via_device_entry = device_registry.async_get_device(
+            {("mqtt", "id_via_very_unique")}
+        )
+        assert via_device_entry is not None
+        assert via_device_entry.name is None
+
+    await hass.async_block_till_done()
+    await hass.async_block_till_done()
+
+    # Now discover the via device (a switch)
+    via_device_config = {
+        "name": None,
+        "command_topic": "test-switch-topic",
+        "unique_id": "very_unique_switch",
+        "device": {"identifiers": ["id_via_very_unique"], "name": "My Switch"},
+    }
+    payload = json.dumps(via_device_config)
+    via_device_discovery_topic = "homeassistant/switch/very_unique/config"
+    async_fire_mqtt_message(
+        hass,
+        via_device_discovery_topic,
+        payload,
+    )
+    await hass.async_block_till_done()
+    await hass.async_block_till_done()
+    via_device_entry = device_registry.async_get_device(
+        {("mqtt", "id_via_very_unique")}
+    )
+    assert via_device_entry is not None
+    assert via_device_entry.name == "My Switch"
+
+    await help_check_discovered_items(hass, device_registry, tag_mock)


Code review comments:
diff_hunk: @@ -1185,6 +1185,24 @@ def device_info_from_specifications(
     return info
 
 
+@callback
+def ensure_via_device_exists(
+    hass: HomeAssistant, device_info: DeviceInfo | None, config_entry: ConfigEntry
+) -> None:
+    """Ensore the via device is in the device registry."""
+    if device_info is None:
path: homeassistant/components/mqtt/entity.py
text: @user1:
```suggestion
    if device_info is None or CONF_VIA_DEVICE not in device_info or device_registry.async_get_device(identifiers={device_info["via_device"]}):
```

@author:
We need to initialize `device_registry` first, we can avoid this call id `device_info` is `None`. Allowed early exit now.

diff_hunk: @@ -1185,6 +1185,24 @@ def device_info_from_specifications(
     return info
 
 
+@callback
+def ensure_via_device_exists(
+    hass: HomeAssistant, device_info: DeviceInfo | None, config_entry: ConfigEntry
+) -> None:
+    """Ensore the via device is in the device registry."""
path: homeassistant/components/mqtt/entity.py
text: @user1:
```suggestion
    """Ensure the via device is in the device registry."""
```