Repository: zulip/zulip

Issue:
Performance issue fetching messages with S3 backend enabled
This is a bug introduced in Zulip 4.0: getting URLs for avatar images is very expensive due to this helper function:

```
In [10]: t = time.time()
    ...: for i in range(1000): x = upload_backend.get_public_upload_url("foo")
    ...: print (time.time() - t)
4.389735460281372
```

This is a regression introduced in https://github.com/zulip/zulip/commit/5a4aecfc40f6c57324a703b74b0b74dc4df3e652

@ryanreh99 @mateuszmandera FYI.

The right fix is to only call the `boto` library function once to get the URL prefix, and then do the rest with URL manipulations.

Once this is fixed, I'll release Zulip 4.4 with the fix, since this is a pretty serious issue.


Discussion:
Hello @zulip/server-misc members, this issue was labeled with the "area: uploads" label, so you may want to check it out!

<!-- areaLabelAddition -->

Resolving patch:
diff --git a/zerver/lib/upload.py b/zerver/lib/upload.py
--- a/zerver/lib/upload.py
+++ b/zerver/lib/upload.py
@@ -386,11 +386,11 @@ def __init__(self) -> None:
         self.avatar_bucket = get_bucket(settings.S3_AVATAR_BUCKET, self.session)
         self.uploads_bucket = get_bucket(settings.S3_AUTH_UPLOADS_BUCKET, self.session)
 
-    def get_public_upload_url(
-        self,
-        key: str,
-    ) -> str:
-        # Return the public URL for a key in the S3 Avatar bucket.
+        self._boto_client = None
+        self.public_upload_url_pattern = self.construct_public_upload_url_pattern()
+
+    def construct_public_upload_url_pattern(self) -> str:
+        # Return the pattern for public URL for a key in the S3 Avatar bucket.
         # For Amazon S3 itself, this will return the following:
         #     f"https://{self.avatar_bucket.name}.{network_location}/{key}"
         #
@@ -399,20 +399,49 @@ def get_public_upload_url(
         # different URL format. Configuring no signature and providing
         # no access key makes `generate_presigned_url` just return the
         # normal public URL for a key.
-        config = Config(signature_version=botocore.UNSIGNED)
-        return self.session.client(
-            "s3",
-            region_name=settings.S3_REGION,
-            endpoint_url=settings.S3_ENDPOINT_URL,
-            config=config,
-        ).generate_presigned_url(
+        #
+        # It unfortunately takes 2ms per query to call
+        # generate_presigned_url, even with our cached boto
+        # client. Since we need to potentially compute hundreds of
+        # avatar URLs in single `GET /messages` request, we instead
+        # back-compute the URL pattern here.
+
+        DUMMY_KEY = "dummy_key_ignored"
+        foo_url = self.get_boto_client().generate_presigned_url(
             ClientMethod="get_object",
             Params={
                 "Bucket": self.avatar_bucket.name,
-                "Key": key,
+                "Key": DUMMY_KEY,
             },
             ExpiresIn=0,
         )
+        parsed = urllib.parse.urlparse(foo_url)
+        base_path = os.path.dirname(parsed.path)
+
+        url_pattern = urllib.parse.urlunparse(
+            parsed._replace(path=os.path.join(base_path, "{key}"))
+        )
+        return url_pattern
+
+    def get_public_upload_url(
+        self,
+        key: str,
+    ) -> str:
+        return self.public_upload_url_pattern.format(key=key)
+
+    def get_boto_client(self) -> botocore.client.BaseClient:
+        """
+        Creating the client takes a long time so we need to cache it.
+        """
+        if self._boto_client is None:
+            config = Config(signature_version=botocore.UNSIGNED)
+            self._boto_client = self.session.client(
+                "s3",
+                region_name=settings.S3_REGION,
+                endpoint_url=settings.S3_ENDPOINT_URL,
+                config=config,
+            )
+        return self._boto_client
 
     def delete_file_from_s3(self, path_id: str, bucket: ServiceResource) -> bool:
         key = bucket.Object(path_id)


Test patch: