internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: WordPress/openverse-api

Issue:
Some waveform peaks have negative values
## Description
<!-- Concisely describe the bug. -->
The frontend expects peaks to be in range of [0..1], however, some peaks returned from the API can have negative values.

## Reproduction
<!-- Provide detailed steps to reproduce the bug. -->
1. Go to https://api.openverse.engineering/v1/audio/727b9979-98f1-4604-9097-0a8b6fa68f2e/
2. Look at the last values of the long array.
3. See error: `...-0.00036,-0.00099,-0.00162,0.00012,0.0]}`

## Expectation
<!-- Concisely describe what you expected to happen. -->
All the peaks should be in [0..1] range.

## Additional context
<!-- Add any other context about the problem here; or delete the section entirely. -->
We could use a hack to fix this on the front end by setting negative values to 0 in the mean time, but the solution should be on the API side.
Could this possibly be some leftovers from the mock waveform data?
## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.



Discussion:
It was inferred from the `audiowaveform` library output that it returns an alternating sequence of positive and negative amplitudes. 

https://github.com/WordPress/openverse-api/blob/4b775952bd3b78ab017f83602225d728d9df39f7/api/catalog/api/utils/waveform.py#L90-L93

But given that it was inferred and not explicitly documented, we should add a floor of zero during normalisation.

The alternative solution (which makes sense in order to avoid re-generating waveforms for all tracks) is, as you mentioned, to set this floor on the frontend. In any case the negative amplitudes are very very close to zero (I'm presuming rounding errors in the `audiowaveform` library) so manipulating them to zero would not be cheating. I'm not against hacks (when they work perfectly well).

Resolving patch:
diff --git a/api/catalog/api/utils/waveform.py b/api/catalog/api/utils/waveform.py
--- a/api/catalog/api/utils/waveform.py
+++ b/api/catalog/api/utils/waveform.py
@@ -106,6 +106,8 @@ def process_waveform_output(json_out):
     for idx, val in enumerate(data):
         if idx % 2 == 0:
             continue
+        if val < 0:  # Any other odd values are negligible and can be ignored
+            val = 0
         transformed_data.append(val)
         if val > max_val:
             max_val = val


Test patch:
Repository: WordPress/openverse-api

Issue:
Return secure URLs for the fields thumbnail, detail_url and related_url.
_(Framed the verbiage of the title as a feature request)_ 🙏 

## Problem

The response for search and detail requests includes insecure URLs (`http` instead of `https`) in the fields `thumbnail`, `detail_url` and `related_url`.

![image](https://user-images.githubusercontent.com/746152/158834288-6b49c399-3b8c-48c8-8529-cdff970686a3.png)


e.g.:

**Search**

https://api.openverse.engineering/v1/images/?q=flower

**Detail:**

https://api.openverse.engineering/v1/images/6c1769b0-a3e5-4dae-8a36-8531a6e1430f/

## Description

When trying to integrate Openverse with some code on the browser I ended up having to replace the scheme part of the URL for avoiding notices like ```xxxx was loaded over HTTPS, but requested an insecure resource 'http://api.openverse.engineering/v1/images/6c1769b0-a3e5-4dae-8a36-8531a6e1430f/'. This request has been blocked; the content must be served over HTTPS.`
 



Discussion:
Code that generates these URLs:

image:
https://github.com/WordPress/openverse-api/blob/4b775952bd3b78ab017f83602225d728d9df39f7/api/catalog/api/serializers/image_serializers.py#L96

audio:
https://github.com/WordPress/openverse-api/blob/4b775952bd3b78ab017f83602225d728d9df39f7/api/catalog/api/serializers/audio_serializers.py#L144

**Update:** these serializer fields work as expected. As Gunicorn sits behind a proxy, it does not know about the SSL setup.
I think this [StackOverflow thread](https://stackoverflow.com/questions/62047354/build-absolute-uri-with-https-behind-reverse-proxy) has the cause of and the solution for this issue. Since our API sits behind a proxy, the Django application must be feeling that it's being served over `http`.

There is a simpler solution, which is just string-replacing `http` in all links to `https`. This might be easier to setup and deploy but feels more like a hack than an actual solution.
Unfortunately our two attempts to address this did not prove successful in production. Re-opening this issue.

Resolving patch:
diff --git a/api/catalog/api/serializers/base.py b/api/catalog/api/serializers/base.py
--- a/api/catalog/api/serializers/base.py
+++ b/api/catalog/api/serializers/base.py
@@ -19,6 +19,6 @@ def get_url(self, *args, **kwargs):
 
         # Only rewrite URLs if a fixed scheme is provided
         if self.scheme is not None:
-            re.sub(r"^\w+://", f"{self.scheme}://", url, 1)
+            url = re.sub(r"^\w+://", f"{self.scheme}://", url, 1)
 
         return url


Test patch:
Repository: WordPress/openverse-api

Issue:
Settings does not respect `DEFAULT_FROM_EMAIL`
## Description
<!-- Concisely describe the bug. Compare your experience with what you expected to happen. -->
<!-- For example: "I clicked the 'submit' button and instead of seeing a thank you message, I saw a blank page." -->
We recently tried setting the `DEFAULT_FROM_EMAIL`[^1] in the environment for the API, but the `settings.py` file doesn't make use of it. We need to add some logic to the `settings.py` file to allow the default to be overridden.

[^1]: https://docs.djangoproject.com/en/4.0/ref/settings/#std:setting-DEFAULT_FROM_EMAIL

## Additional context
<!-- Add any other context about the problem here; or delete the section entirely. -->
Blocks the resolution of #506

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -298,6 +298,7 @@
 EMAIL_HOST_PASSWORD = config("EMAIL_HOST_PASSWORD", default="")
 EMAIL_SUBJECT_PREFIX = "[noreply]"
 EMAIL_USE_TLS = True
+DEFAULT_FROM_EMAIL = config("DEFAULT_FROM_EMAIL", default="")
 
 if EMAIL_HOST_USER or EMAIL_HOST_PASSWORD:
     EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"


Test patch:
Repository: WordPress/openverse-api

Issue:
Possibly make `thumbnail` null for audio files without artwork
## Description
<!-- Concisely describe the bug. -->

Currently the frontend tries to fetch thumbnails for all audio files regardless of whether the audio file in question has one or not. 
I noticed that the API returns the thumbnail URL for all tracks. That makes sense, but could we improve this to be `null` for audio tracks without artwork? Then we could check the field in the frontend before making a network request.


Discussion:
The thumbnail endpoint does this:
- if the audio has any artwork, return the URL to that
- if the audio's audio-set has any artwork, return the URL to that

This means that on the search results page, if we try to validate if the audio has artwork if will cause N lookups for the audio sets associated with each of the N audio results (similar problem as #524). We can mitigate this issue by prefetching the sets with the audio.

Resolving patch:
diff --git a/api/catalog/api/serializers/audio_serializers.py b/api/catalog/api/serializers/audio_serializers.py
--- a/api/catalog/api/serializers/audio_serializers.py
+++ b/api/catalog/api/serializers/audio_serializers.py
@@ -135,6 +135,18 @@ def get_peaks(obj) -> list[int]:
             obj = Audio.objects.get(identifier=obj.identifier)
         return obj.get_waveform()
 
+    def to_representation(self, instance):
+        # Get the original representation
+        output = super().to_representation(instance)
+
+        if isinstance(instance, Hit):
+            # TODO: Remove when updating ES indexes
+            audio = Audio.objects.get(identifier=instance.identifier)
+            if not audio.thumbnail:
+                output["thumbnail"] = None
+
+        return output
+
 
 class AudioSearchSerializer(MediaSearchSerializer):
     """


Test patch:
diff --git a/api/test/audio_integration_test.py b/api/test/audio_integration_test.py
--- a/api/test/audio_integration_test.py
+++ b/api/test/audio_integration_test.py
@@ -81,6 +81,14 @@ def test_audio_thumb(audio_fixture):
     thumb(audio_fixture)
 
 
+def test_audio_without_thumb():
+    """The first audio of this search should not have a thumbnail."""
+    resp = requests.get(f"{API_URL}/v1/audio?q=zaus&filter_dead=false")
+    assert resp.status_code == 200
+    parsed = json.loads(resp.text)
+    assert parsed["results"][0]["thumbnail"] is None
+
+
 def test_audio_thumb_compression(audio_fixture):
     thumb_compression(audio_fixture)
Repository: WordPress/openverse-api

Issue:
Thumbnail timeouts raise `TimeoutError`
## Description
<!-- Concisely describe the bug. Compare your experience with what you expected to happen. -->
<!-- For example: "I clicked the 'submit' button and instead of seeing a thank you message, I saw a blank page." -->
We recently changed the thumbnail service we're using and some of the business logic around it (#630). The previous thumbnail retrieval step [did not have a timeout](https://github.com/WordPress/openverse-api/blame/0df21059dfe579be1e7a83f64e8e8a7d4e422416/api/catalog/api/views/media_views.py#L156), whereas the new thumbnail retrieval step has a timeout of 5 seconds:

https://github.com/WordPress/openverse-api/blob/03362db8caed8014e31d3fe5274c9bb084086b85/api/catalog/api/views/media_views.py#L173

We should also add a case for the `TimeoutError` exception here:

https://github.com/WordPress/openverse-api/blob/03362db8caed8014e31d3fe5274c9bb084086b85/api/catalog/api/views/media_views.py#L183

I think it might also be best for us to increase this timeout just a bit (maybe 10s?) to give the thumbnails for larger images/museum images more time to load.

## Additional context
<!-- Add any other context about the problem here; or delete the section entirely. -->
Sentry issue: https://sentry.io/share/issue/3daf5a2129a449859200e2de0c458ed6/

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.



Discussion:
It seems we may also want to add [`RemoteDisconnected` exceptions as well](https://sentry.io/share/issue/7798086df599457ca6670e31381901bc/)
Perhaps we should have a broader exception clause in general? Here's a [`ConnectionResetError` failure](https://sentry.io/share/issue/9282ce4f2f5049ccb8b346bb48f225a3/)

Resolving patch:
diff --git a/api/catalog/api/views/media_views.py b/api/catalog/api/views/media_views.py
--- a/api/catalog/api/views/media_views.py
+++ b/api/catalog/api/views/media_views.py
@@ -1,5 +1,6 @@
 import json
 import logging as log
+from http.client import RemoteDisconnected
 from urllib.error import HTTPError
 from urllib.parse import urlencode
 from urllib.request import Request, urlopen
@@ -170,7 +171,7 @@ def _thumbnail_proxy_comm(
             req = Request(upstream_url)
             for key, val in headers:
                 req.add_header(key, val)
-            upstream_response = urlopen(req, timeout=5)
+            upstream_response = urlopen(req, timeout=10)
 
             res_status = upstream_response.status
             content_type = upstream_response.headers.get("Content-Type")
@@ -180,8 +181,12 @@ def _thumbnail_proxy_comm(
             )
 
             return upstream_response, res_status, content_type
-        except HTTPError as exc:
+        except (HTTPError, RemoteDisconnected, TimeoutError) as exc:
             raise get_api_exception(f"Failed to render thumbnail: {exc}")
+        except Exception as exc:
+            raise get_api_exception(
+                f"Failed to render thumbnail due to unidentified exception: {exc}"
+            )
 
     @staticmethod
     def _get_proxied_image(


Test patch:
Repository: WordPress/openverse-api

Issue:
Reduce throttle across all endpoints
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
We continue to see what appears to be abusive behavior and attempts to circumvent throttling by using distributed requests at just below the throttling level.

## Description
<!-- Describe the feature and how it solves the problem. -->
Given that it is trivial to register for authorization to use our API at normal limits through our OAuth flow, we can reduce the throttling rate to a very low value. 1/day is far too restrictive. 1/hour was suggested by @AetherUnbound but I think that's still probably too high as it will interrupt the ability for someone to easily use the API.

Maybe something like 5/hour would still allow for relatively easy debugging but still making it harder to create a distributed request pattern that causes such high resource utilization.

## Alternatives
<!-- Describe any alternative solutions or features you have considered. How is this feature better? -->
Require authorization to make any API requests at all.

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:

Resolving patch:
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -118,6 +118,9 @@
 
 OAUTH2_PROVIDER_APPLICATION_MODEL = "api.ThrottledApplication"
 
+THROTTLE_ANON_BURST = config("THROTTLE_ANON_BURST", default="5/hour")
+THROTTLE_ANON_SUSTAINED = config("THROTTLE_ANON_SUSTAINED", default="100/day")
+
 REST_FRAMEWORK = {
     "DEFAULT_AUTHENTICATION_CLASSES": (
         "oauth2_provider.contrib.rest_framework.OAuth2Authentication",
@@ -137,8 +140,8 @@
         "catalog.api.utils.throttle.EnhancedOAuth2IdThrottleBurstRate",
     ),
     "DEFAULT_THROTTLE_RATES": {
-        "anon_burst": "60/min",
-        "anon_sustained": "5000/day",
+        "anon_burst": THROTTLE_ANON_BURST,
+        "anon_sustained": THROTTLE_ANON_SUSTAINED,
         "oauth2_client_credentials_sustained": "10000/day",
         "oauth2_client_credentials_burst": "100/min",
         "enhanced_oauth2_client_credentials_sustained": "20000/day",


Test patch:
Repository: WordPress/openverse-api

Issue:
Update the auth token expiry to be 12 or 24 hours
## Problem
Currently the token returned from the auth_tokens/token route returns a token that expires in 10 hours. To make this a little more consistent with typical CRON scheduling it would be more ideal for this expiry to be 12 or 24 hours.

## Description
Updating the auth token expires_in time would allow for more consistent CRON scheduling and most users would more typically expect it to expire every 12/24 hours.

## Alternatives
No suggestions aside from even longer expiry times (maybe once a year or once a month, etc).

## Additional context
Issue/suggestion originally came up here: [https://github.com/WordPress/openverse-api/issues/770](https://github.com/WordPress/openverse-api/issues/770)

## Implementation
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:
Can I work on this?
Yes please @raghuvar-arora-au2! I've assigned the issue to you. Thank you!!

Resolving patch:
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -113,7 +113,10 @@
     "SCOPES": {
         "read": "Read scope",
         "write": "Write scope",
-    }
+    },
+    "ACCESS_TOKEN_EXPIRE_SECONDS": config(
+        "ACCESS_TOKEN_EXPIRE_SECONDS", default=3600 * 12, cast=int
+    ),
 }
 
 OAUTH2_PROVIDER_APPLICATION_MODEL = "api.ThrottledApplication"


Test patch:
Repository: WordPress/openverse-api

Issue:
Make `ALLOWED_HOSTS` configurable via environment
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
Certain environments should be able to have more relaxed or expansive allowed hosts rules (like allowing local env to receive requests from other docker networks through `host.docker.internal` or the like). Likewise, alternative deployments of the Openverse API would have to manually change this anyway because we currently hard code our `openverse.engineering` addresses.

## Description
<!-- Describe the feature and how it solves the problem. -->
Make `ALLOWED_HOSTS` configurable via environment variable. It should handle a comma separated string.

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:

Resolving patch:
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -44,9 +44,7 @@
 
 ENVIRONMENT = config("ENVIRONMENT", default="local")
 
-ALLOWED_HOSTS = [
-    "api-dev.openverse.engineering",
-    "api.openverse.engineering",
+ALLOWED_HOSTS = config("ALLOWED_HOSTS").split(",") + [
     gethostname(),
     gethostbyname(gethostname()),
 ]


Test patch:
Repository: WordPress/openverse-api

Issue:
Remove Sentry DSN from repository
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
If someone configures their local environment correctly, they can send scary (and hard to discern) errors to Sentry that we can easily mistake for production issues. It's not likely that anyone is doing this intentionally (what do they stand to gain, other than trolling, I suppose) but it is easy to do accidentally (just turn off the Django DEBUG setting).

## Description
<!-- Describe the feature and how it solves the problem. -->
Let's just remove the Sentry DSN from the repository. The environment variable can default to empty and Sentry just won't be enabled in that case. In production and staging we'll pass the Sentry DSN like we do any other secret. Let's go ahead and rotate the DSN as well once we deploy this change.

## Alternatives
Make it harder to accidentally (or intentionally) configure Sentry to run locally. Maybe Sentry has an optional password or something.

## Additional context
Marked as Staff only as it requires infrastructural changes to be completed.

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:

Resolving patch:
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -337,13 +337,11 @@
 # https://docs.djangoproject.com/en/4.0/releases/4.0/#csrf-trusted-origins-changes-4-0
 CSRF_TRUSTED_ORIGINS = ["https://*.openverse.engineering"]
 
-SENTRY_DSN = config(
-    "SENTRY_DSN",
-    default="https://08f4706d16004f57bcd37eb907bfc2e7@o787041.ingest.sentry.io/6107216",
-)
+SENTRY_DSN = config("SENTRY_DSN", default="")
+
 SENTRY_SAMPLE_RATE = config("SENTRY_SAMPLE_RATE", default=1.0, cast=float)
 
-if not DEBUG:
+if not DEBUG and SENTRY_DSN:
     sentry_sdk.init(
         dsn=SENTRY_DSN,
         integrations=[DjangoIntegration()],


Test patch:
Repository: WordPress/openverse-api

Issue:
Use `terms` instead of `term`
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
In the search controller, when performing filtering for more than one value, we use an array of `term` queries instead of using `terms`. 

https://github.com/WordPress/openverse-api/blob/6cc0551c639081f895d974b30cc465713d0ce360/api/catalog/api/controllers/search_controller.py#L172-L177

## Description
Replace the array of `term` queries with a singular `term` query. According to the [ES docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html), the `terms` query is functionally identical to `term`.

## Additional context
<!-- Add any other context about the feature here; or delete the section entirely. -->
For an example, a `terms` query is used just a few lines below in the same file, inside the `_exclude_filtered` function.

https://github.com/WordPress/openverse-api/blob/6cc0551c639081f895d974b30cc465713d0ce360/api/catalog/api/controllers/search_controller.py#L197

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:
raising_hand I would be interested in implementing this feature.

Resolving patch:
diff --git a/api/catalog/api/controllers/search_controller.py b/api/catalog/api/controllers/search_controller.py
--- a/api/catalog/api/controllers/search_controller.py
+++ b/api/catalog/api/controllers/search_controller.py
@@ -241,15 +241,16 @@ def _apply_filter(
     """
 
     if serializer_field in search_params.data:
-        filters = []
-        for arg in search_params.data[serializer_field].split(","):
-            _param = es_field or serializer_field
-            args = {"name_or_query": "term", _param: arg}
-            filters.append(Q(**args))
+        arguments = search_params.data.get(serializer_field)
+        if arguments is None:
+            return s
+        arguments = arguments.split(",")
+        parameter = es_field or serializer_field
+        query = Q("terms", **{parameter: arguments})
         method = getattr(s, behaviour)
-        return method("bool", should=filters)
-    else:
-        return s
+        return method("bool", should=query)
+
+    return s
 
 
 def _exclude_filtered(s: Search):


Test patch:
Repository: WordPress/openverse-api

Issue:
Unhandled value out of range
## Sentry link
<!-- The public (aka "share") Sentry issue link. -->
https://sentry.io/share/issue/7dd836d27daa41dfa59dfbdf39a40d51/

## Description
<!-- Example: We are trying to access property foo of ImportantClass but the instance is null. -->
<!-- Include any additional information you may have, including potential remedies if any come to mind, and the general context of the code (what causes it to run in the app).  -->

We need to capture and handle this error to avoid returning a 500 error in the link below.

> 'l' format requires -2147483648 <= number <= 2147483647

Originates from this fragment of code:

https://github.com/WordPress/openverse-api/blob/81b7860e2662f4b135919da0728eea194f15c260/api/catalog/api/views/image_views.py#L125-L130

<!-- Mention whether this is a known regression, i.e., the feature used to work and now does not.  -->

## Reproduction
<!-- Share the steps to reproduce the issue, if you were able to, OR a note sharing that you tried to reproduce but weren’t able to. -->
1. Go to https://api-dev.openverse.engineering/v1/images/6c855680-b426-40f1-adc6-f61f434d0be9/watermark/

## Aditional context
Related to #483.



Discussion:

Resolving patch:
diff --git a/api/catalog/api/views/image_views.py b/api/catalog/api/views/image_views.py
--- a/api/catalog/api/views/image_views.py
+++ b/api/catalog/api/views/image_views.py
@@ -1,4 +1,5 @@
 import io
+import struct
 
 from django.conf import settings
 from django.http.response import FileResponse, HttpResponse
@@ -131,7 +132,14 @@ def watermark(self, request, *_, **__):
         watermarked, exif = watermark(image_url, image_info, params.data["watermark"])
         # Re-insert EXIF metadata.
         if exif:
-            exif_bytes = piexif.dump(exif)
+            # piexif dump raises InvalidImageDataError which is a child class
+            # of ValueError, and a struct error when the value is not
+            # between -2147483648 and 2147483647
+            # https://github.com/WordPress/openverse-api/issues/849
+            try:
+                exif_bytes = piexif.dump(exif)
+            except (struct.error, ValueError):
+                exif_bytes = None
         else:
             exif_bytes = None
         img_bytes = io.BytesIO()


Test patch: