Repository: readthedocs/readthedocs.org

Issue:
Notifications: duplicate messages
Noticed here:
https://beta.readthedocs.org/projects/xclim/builds/23494705/

The error messages seem to stack here:

![image](https://github.com/readthedocs/readthedocs.org/assets/1140183/d7aee802-1329-4a5f-97a4-f9bb2bfb087c)

Also not sure why the string interpolation fails for these, I've noticed that in a few spots though.

On the current dashboard there are no notifications though, not sure why.

https://readthedocs.org/projects/xclim/builds/23494705/


Discussion:
cc @humitos I don't know if we've fixed this yet or not
Without jumping too much on this yet:

1. for some reason the build is not being reset properly when it's retried. We have code that cleanup all the notifications: https://github.com/readthedocs/readthedocs.org/blob/0a381a7acfc7683755ad0af3ad652c3b2f73fe25/readthedocs/builds/models.py#L1092-L1109
2. since the notification is exactly the same, it should be de-duplicated when it's added. However, when checking the APIv2 view (the one used by the builders), it doesn't seem we are using our custom `.add()` method that de-duplicates notifications: https://github.com/readthedocs/readthedocs.org/blob/0a381a7acfc7683755ad0af3ad652c3b2f73fe25/readthedocs/api/v2/views/model_views.py#L382-L399

So, working on 2) to call `.add()` should be the first point to attack here. I suppose we should overwrite the `api.v2.NotificationSerializer.save()` method to call our custom `.add()`.
> On the current dashboard there are no notifications though, not sure why.
> [readthedocs.org/projects/xclim/builds/23494705](https://readthedocs.org/projects/xclim/builds/23494705/)

I think this is because we implemented permissions on notifications in the last deploy (see https://github.com/readthedocs/readthedocs.org/pull/11117) and there may be a bug on it because those notifications should be shown to all the users, since the build is public.
> Also not sure why the string interpolation fails for these, I've noticed that in a few spots though.

```python
In [12]: Notification.objects.filter(message_id=BuildMaxConcurrencyError.LIMIT_REACHED, format_values={}).count()
Out[12]: 4595

In [13]: Notification.objects.filter(message_id=BuildMaxConcurrencyError.LIMIT_REACHED).exclude(format_values={}).count()
Out[13]: 43746
```

It seems somewhere we are creating this notification without the proper `format_values`. 10% of them are broken. I suppose there should be a Celery workflow that we are not handling correctly here. I doubt it's this one in particular https://github.com/readthedocs/readthedocs.org/blob/0a381a7acfc7683755ad0af3ad652c3b2f73fe25/readthedocs/projects/tasks/builds.py#L362-L374
This is probably worth a separate issue, but I've noticed these build concurrency notifications on a number of builds too. Shouldn't these notifications clear after the build is successfully started?

![image](https://github.com/readthedocs/readthedocs.org/assets/1140183/99d0e1b1-bf8d-43b0-b78c-ee36be2c730a)

I've noticed these in a few spots, but it's hard for me to tell if this should be resolved now or not. The above error is on my local instance on a finished build -- same with the original errors I posted here.

I suppose also these should be warning level notifications? That would at least help make it clear that the build is not failed.
> I suppose also these should be warning level notifications? That would at least help make it clear that the build is not failed.

I've done this in https://github.com/readthedocs/readthedocs.org/pull/11196
> Shouldn't these notifications clear after the build is successfully started?

Yes, I've mentioned this in 1) from my previous comment https://github.com/readthedocs/readthedocs.org/issues/11131#issuecomment-1956118651
> So, working on 2) to call `.add()` should be the first point to attack here. I suppose we should overwrite the `api.v2.NotificationSerializer.save()` method to call our custom `.add()`.

This is done in https://github.com/readthedocs/readthedocs.org/pull/11197
Now, with the two PRs I've opened I think this issue shouldn't be an issue anymore. However, 1) could be still present sometimes due to network issues/web instance congestion or similar scenarios where the `reset` API call fails to perform. That said, I'd would move forward with what we have keeping the issue open for now in case we continue seeing it but I wouldn't invest too much time on digging further on debugging for now.
> Yes, I've mentioned this in 1) from my previous comment https://github.com/readthedocs/readthedocs.org/issues/11131#issuecomment-1956118651

That comment implied to me that the messages would only be _deduplicated_ with the fix -- ie:

> since the notification is exactly the same, it should be de-duplicated when it's added. 

Are you saying that this fix is also what will dismiss the notification once the build has started?

To clarify, what I would expect here is that while the build is still queued that warning message should show, but once the build is no longer queued due to concurrency limits the message will disappear.
> what I would expect here is that while the build is still queued that warning message should show, but once the build is no longer queued due to concurrency limits the message will disappear.

Yes, this is how it should work.

However, even with all these fixes... the scenario I mentioned before, may be still present on some edge cases 👇🏼 

>> However, 1) could be still present sometimes due to network issues/web instance congestion or similar scenarios where the `reset` API call fails to perform.
 
I found this issue in https://readthedocs.org/projects/docs/builds/23722719/ as well. This was a recent build in our own documentation. I still don't understand why this is happening, but it's worth investigating it a little more.
We should probably run a small Python code from Django shell to remove these notifications from builds that are in finished state.

Resolving patch:
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
--- a/readthedocs/projects/tasks/builds.py
+++ b/readthedocs/projects/tasks/builds.py
@@ -447,10 +447,14 @@ def before_start(self, task_id, args, kwargs):
         self._reset_build()
 
     def _reset_build(self):
-        # Reset build only if it has some commands already.
-        if self.data.build.get("commands"):
-            log.info("Resetting build.")
-            self.data.api_client.build(self.data.build["id"]).reset.post()
+        # Always reset the build before starting.
+        # We used to only reset it when it has at least one command executed already.
+        # However, with the introduction of the new notification system,
+        # it could have a notification attached (e.g. Max concurrency build)
+        # that needs to be removed from the build.
+        # See https://github.com/readthedocs/readthedocs.org/issues/11131
+        log.info("Resetting build.")
+        self.data.api_client.build(self.data.build["id"]).reset.post()
 
     def on_failure(self, exc, task_id, args, kwargs, einfo):
         """


Test patch:
diff --git a/readthedocs/projects/tests/mockers.py b/readthedocs/projects/tests/mockers.py
--- a/readthedocs/projects/tests/mockers.py
+++ b/readthedocs/projects/tests/mockers.py
@@ -208,6 +208,11 @@ def _mock_api(self):
             status_code=201,
         )
 
+        self.requestsmock.post(
+            f"{settings.SLUMBER_API_HOST}/api/v2/build/{self.build.pk}/reset/",
+            status_code=201,
+        )
+
         self.requestsmock.get(
             f"{settings.SLUMBER_API_HOST}/api/v2/build/concurrent/?project__slug={self.project.slug}",
             json=lambda request, context: {
diff --git a/readthedocs/projects/tests/test_build_tasks.py b/readthedocs/projects/tests/test_build_tasks.py
--- a/readthedocs/projects/tests/test_build_tasks.py
+++ b/readthedocs/projects/tests/test_build_tasks.py
@@ -297,9 +297,9 @@ def test_build_updates_documentation_type(self, load_yaml_config):
         self._trigger_update_docs_task()
 
         # Update version state
-        assert self.requests_mock.request_history[7]._request.method == "PATCH"
-        assert self.requests_mock.request_history[7].path == "/api/v2/version/1/"
-        assert self.requests_mock.request_history[7].json() == {
+        assert self.requests_mock.request_history[8]._request.method == "PATCH"
+        assert self.requests_mock.request_history[8].path == "/api/v2/version/1/"
+        assert self.requests_mock.request_history[8].json() == {
             "addons": False,
             "build_data": None,
             "built": True,
@@ -477,8 +477,12 @@ def test_successful_build(
 
         # TODO: assert the verb and the path for each API call as well
 
+        # Build reset
+        assert self.requests_mock.request_history[3]._request.method == "POST"
+        assert self.requests_mock.request_history[3].path == "/api/v2/build/1/reset/"
+
         # Update build state: cloning
-        assert self.requests_mock.request_history[3].json() == {
+        assert self.requests_mock.request_history[4].json() == {
             "id": 1,
             "state": "cloning",
             "commit": "a1b2c3",
@@ -487,7 +491,7 @@ def test_successful_build(
         }
 
         # Update build state: installing
-        assert self.requests_mock.request_history[4].json() == {
+        assert self.requests_mock.request_history[5].json() == {
             "id": 1,
             "state": "installing",
             "commit": "a1b2c3",
@@ -550,7 +554,7 @@ def test_successful_build(
             },
         }
         # Update build state: building
-        assert self.requests_mock.request_history[5].json() == {
+        assert self.requests_mock.request_history[6].json() == {
             "id": 1,
             "state": "building",
             "commit": "a1b2c3",
@@ -560,7 +564,7 @@ def test_successful_build(
             "error": "",
         }
         # Update build state: uploading
-        assert self.requests_mock.request_history[6].json() == {
+        assert self.requests_mock.request_history[7].json() == {
             "id": 1,
             "state": "uploading",
             "commit": "a1b2c3",
@@ -570,9 +574,9 @@ def test_successful_build(
             "error": "",
         }
         # Update version state
-        assert self.requests_mock.request_history[7]._request.method == "PATCH"
-        assert self.requests_mock.request_history[7].path == "/api/v2/version/1/"
-        assert self.requests_mock.request_history[7].json() == {
+        assert self.requests_mock.request_history[8]._request.method == "PATCH"
+        assert self.requests_mock.request_history[8].path == "/api/v2/version/1/"
+        assert self.requests_mock.request_history[8].json() == {
             "addons": False,
             "build_data": None,
             "built": True,
@@ -582,11 +586,11 @@ def test_successful_build(
             "has_htmlzip": True,
         }
         # Set project has valid clone
-        assert self.requests_mock.request_history[8]._request.method == "PATCH"
-        assert self.requests_mock.request_history[8].path == "/api/v2/project/1/"
-        assert self.requests_mock.request_history[8].json() == {"has_valid_clone": True}
+        assert self.requests_mock.request_history[9]._request.method == "PATCH"
+        assert self.requests_mock.request_history[9].path == "/api/v2/project/1/"
+        assert self.requests_mock.request_history[9].json() == {"has_valid_clone": True}
         # Update build state: finished, success and builder
-        assert self.requests_mock.request_history[9].json() == {
+        assert self.requests_mock.request_history[10].json() == {
             "id": 1,
             "state": "finished",
             "commit": "a1b2c3",
@@ -598,8 +602,8 @@ def test_successful_build(
             "error": "",
         }
 
-        assert self.requests_mock.request_history[10]._request.method == "POST"
-        assert self.requests_mock.request_history[10].path == "/api/v2/revoke/"
+        assert self.requests_mock.request_history[11]._request.method == "POST"
+        assert self.requests_mock.request_history[11].path == "/api/v2/revoke/"
 
         assert BuildData.objects.all().exists()