Repository: fedora-infra/bodhi

Issue:
bodhi fails to contact koji in prod
````
curl "https://bodhi.fedoraproject.org/latest_candidates?hide_existing=true&prefix=shutter"
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
````

Since bodhi 5.1 this is no longer a blocker but it's also definitely not a nice UX.


Originally reported at: https://pagure.io/fedora-infrastructure/issue/8451


Discussion:
mmm looking at koji.listTagged code:
https://github.com/koji-project/koji/blob/2460a00117f8f7c9a64c96415cf638d9385a824b/hub/kojihub.py#L9772-L9780

I think that our request to koji simply hits timeout because koji firstly retrieves ALL builds and then it uses prefix to filter out results... and we're requesting all builds tagged with the candidate tag for all active releases!

In the old update form we were using the package name to filter them out and that was a lot friendlier towards koji resources... I don't think we can use the koji call this way. It would require koji guys to change how the `prefix` filter is applied in their code.
On Mon, Dec 09, 2019 at 09:39:53AM -0800, Mattia Verga wrote:
>    mmm looking at koji.listTagged code:
>    https://github.com/koji-project/koji/blob/2460a00117f8f7c9a64c96415cf638d9385a824b/hub/kojihub.py#L9772-L9780
> 
>    I think that our request to koji simply hits timeout because koji firstly
>    retrieves ALL builds and then it uses prefix to filter out results... and
>    we're requesting all builds tagged with the candidate tag for all active
>    releases!
> 
>    In the old update form we were using the package name to filter them out
>    and that was a lot friendlier towards koji resources... I don't think we
>    can use the koji call this way. It would require koji guys to change how
>    the prefix filter is applied in their code.

IIUC we do both, first one large call then we restrict by the data the user
input.
Maybe we could drop the first call?
I also find the library to trigger very quickly, if you look at your network tab
while typing something like g.u.a.k.e, depending on your speed you may end up
doing up to 5 requests.

Interesting, yesterday I did some tests and every request for latest_candidates with the `prefix` parameter returned timeout.
Today I get the timeout with
https://bodhi.stg.fedoraproject.org/latest_candidates?hide_existing=true&prefix=
which is the query triggered when entering the update form, used to pre-populate the list of available builds, but I can get a response without the `hide_existing` parameter (or by set it to false):
https://bodhi.stg.fedoraproject.org/latest_candidates?prefix=

In both cases the request to koji is the same, so it seems that the timeout is not caused by koji, but it's on our side.
This is the relevant code where we use `hide_existing`: https://github.com/fedora-infra/bodhi/blob/607acaf76dee957791a2dfbe699004102c4cccec/bodhi/server/views/generic.py#L289-L370

In https://github.com/fedora-infra/bodhi/commit/ce9ec304837fd1f85687eed0d26a83369a69e941#diff-70cb964009a5757f62cddda8f04b3118 a database query has been added without filtering for archived releases, maybe it's too heavy? Adding @nphilipp to the discussion, since he made that commit.
When exactly the `candidate_tag` is removed from a build? Does it happens when it gets tagged `pending_signing_tag`, `pending_testing_tag` or only when it's finally tagged with `testing_tag`?
If the latter, we don't need to query for all builds associated to an update, we only need to filter out all builds from updates in `pending_testing`. Unless we don't use `hide_existing` together with `testing`, but that doesn't make sense to me (we should also avoid that).

Does that sounds right to you?

Resolving patch:
diff --git a/bodhi/server/views/generic.py b/bodhi/server/views/generic.py
--- a/bodhi/server/views/generic.py
+++ b/bodhi/server/views/generic.py
@@ -299,6 +299,9 @@ def work(testing, hide_existing, pkg=None, prefix=None):
 
         if hide_existing:
             # We want to filter out builds associated with an update.
+            # Since the candidate_tag is removed when an update is pushed to
+            # stable, we only need a list of builds that are associated to
+            # updates still in pending state.
 
             # Don't filter by releases here, because the associated update
             # might be archived but the build might be inherited into an active
@@ -306,7 +309,9 @@ def work(testing, hide_existing, pkg=None, prefix=None):
             # this set should be easy enough.
             associated_build_nvrs = set(
                 row[0] for row in
-                db.query(models.Build.nvr).filter(models.Build.update_id != None)
+                db.query(models.Build.nvr).
+                join(models.Update).
+                filter(models.Update.status == models.UpdateStatus.pending)
             )
 
         kwargs = dict(package=pkg, prefix=prefix, latest=True)


Test patch: