internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mosaicml/composer

Issue:
HuggingFace model should update word embeddings dimension according to tokenizer
** To reproduce

Steps to reproduce the behavior:

Right now if I call 

```
model = transformers.AutoModelForCausalLM.from_pretrained('bigscience/bloom-560m')
tokenizer = transformers.AutoTokenizer.from_pretrained('gpt2')

model = HuggingFaceModel(model=model, tokenizer=tokenizer)
```

The bloom model will have a [250880, 1024] dim word embedding matrix and a vocab size of 250880 even though a gpt2 tokenizer with vocab size of 50257 is used.

## Expected behavior

The class HuggingFaceModel needs to update word embedding matrix if a tokenizer is supplied.




Discussion:

Resolving patch:
diff --git a/composer/models/huggingface.py b/composer/models/huggingface.py
--- a/composer/models/huggingface.py
+++ b/composer/models/huggingface.py
@@ -5,6 +5,7 @@
 
 from __future__ import annotations
 
+import logging
 from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
 
 from torchmetrics import Metric
@@ -15,6 +16,8 @@
 if TYPE_CHECKING:
     import transformers
 
+log = logging.getLogger(__name__)
+
 __all__ = ['HuggingFaceModel']
 
 
@@ -66,6 +69,13 @@ def __init__(self,
             assert tokenizer.model_input_names is not None, 'the tokenizer should have a model input name'
             self.model_inputs = set(tokenizer.model_input_names)
 
+            if self.config.vocab_size != len(tokenizer):
+                # set model's word embedding matrix and final lm_head to vocab size according to tokenizer
+                log.warning(
+                    f'The number of tokens in the tokenizer and the number of tokens in the model are different.'
+                    f' Resizing the model tokenizer to {len(tokenizer)} from {self.config.vocab_size}.')
+                self.model.resize_token_embeddings(len(tokenizer))
+
         self.use_logits = use_logits
 
         self.train_metrics = None


Test patch:
Repository: mosaicml/composer

Issue:
Less strict numpy version pinning
## 🚀 Feature Request
Allow for newer numpy versions than `<1.23.0`.

## Motivation

Currently, composer fixes numpy to be `'numpy>=1.21.5,<1.23.0'`. This is unfortunate, because other requirements that we use need numpy > 1.23.0, creating an incompatibility.

This was set in #1345 in reaction to a build failure because scipy need(ed) `<1.23.0`. Scipy itself is apparently pulled from `torch-metrics`. 

I don't believe this is current anymore:
- composer pulls in `torchmetrics>=0.10.0,<0.11.4`
- torchmetrics 0.11.3 pulls in `scipy >1.0.0, <=1.10.0` [(requirements)](https://github.com/Lightning-AI/metrics/blob/v0.11.3/requirements/image.txt#LL4C23-L4C23)
- scipy 1.10.0 sets `np_maxversion = '1.27.0'` [(setup.py)](https://github.com/scipy/scipy/blob/v1.10.0/setup.py#L453)

Indeed, scipy has https://github.com/scipy/scipy/issues/16964, which was fixed in https://github.com/scipy/scipy/pull/16966 in September of 2022, while #1345 above was noted in August.

Accordingly, it looks like relaxing the numpy maxversion should have a valid requirements path. 

The closed PR https://github.com/mosaicml/composer/pull/1835 would already implement this.


Discussion:
Hey @fgerzer thanks for raising this! Good chance we can safely upgrade numpy; will test it out.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -80,7 +80,7 @@ def package_files(prefix: str, directory: str, extension: str):
     'torchvision>=0.11.0,<0.15',
     'torch>=1.10.0,<1.14',
     'requests>=2.26.0,<3',
-    'numpy>=1.21.5,<1.23.0',
+    'numpy>=1.21.5,<1.25.0',
     'psutil>=5.8.0,<6',
     'coolname>=1.1.0,<3',
     'tabulate==0.9.0',  # for auto-generating tables


Test patch:
diff --git a/tests/trainer/test_sharded_checkpoint.py b/tests/trainer/test_sharded_checkpoint.py
--- a/tests/trainer/test_sharded_checkpoint.py
+++ b/tests/trainer/test_sharded_checkpoint.py
@@ -105,7 +105,7 @@ def test_fsdp_full_state_dict_save(world_size, tmp_path: pathlib.Path):
 
     expected_layer_shapes = [(5, num_features), (5,), (num_classes, 5), (num_classes,)]
     layer1_weights_shape, layer1_bias_shape, layer2_weights_shape, layer2_bias_shape = expected_layer_shapes
-    expected_total_num_params = sum([np.prod(shape) for shape in expected_layer_shapes])
+    expected_total_num_params = sum([np.prod(shape) for shape in expected_layer_shapes])  # type: ignore
 
     trainer = get_trainer(save_folder=str(save_folder),
                           save_filename=save_filename,
@@ -230,7 +230,7 @@ def test_fsdp_partitioned_state_dict_save(world_size, tmp_path: pathlib.Path, st
     num_classes = 2
 
     expected_layer_shapes = [(5, num_features), (5,), (num_classes, 5), (num_classes,)]
-    expected_total_num_params = sum([np.prod(shape) for shape in expected_layer_shapes])
+    expected_total_num_params = sum([np.prod(shape) for shape in expected_layer_shapes])  # type: ignore
 
     trainer = get_trainer(save_folder=str(save_folder),
                           save_filename=save_filename,
Repository: WordPress/openverse-api

Issue:
Replace deprecated falcon.api with falcon.app
> The API class was renamed to App in Falcon 3.0. The old class name remains available as an alias for backwards-compatibility, but will be removed in a future release.

[Falcon docs](https://falcon.readthedocs.io/en/latest/api/app.html)

We use `falcon` in `analytics` and in the `ingestion_server`. We should replace deprecated `app` class name with `api` class name.


Discussion:

Resolving patch:
diff --git a/analytics/server.py b/analytics/server.py
--- a/analytics/server.py
+++ b/analytics/server.py
@@ -83,7 +83,7 @@ def on_get(self, req, resp):
     allow_all_methods=True,
     allow_all_headers=True
 )
-api = falcon.API(middleware=[cors.middleware])
+api = falcon.App(middleware=[cors.middleware])
 api.add_route('/', RedocResource())
 api.add_route('/swagger.yaml', OpenAPISpecResource())
 api.add_route('/search_event', SearchEventResource())
diff --git a/ingestion_server/ingestion_server/api.py b/ingestion_server/ingestion_server/api.py
--- a/ingestion_server/ingestion_server/api.py
+++ b/ingestion_server/ingestion_server/api.py
@@ -169,7 +169,7 @@ def create_api(log=True):
         handler.setFormatter(formatter)
         root.addHandler(handler)
 
-    _api = falcon.API()
+    _api = falcon.App()
     task_tracker = TaskTracker()
     task_resource = TaskResource(task_tracker)
     get_task_status = TaskStatus(task_tracker)
diff --git a/ingestion_server/ingestion_server/indexer_worker.py b/ingestion_server/ingestion_server/indexer_worker.py
--- a/ingestion_server/ingestion_server/indexer_worker.py
+++ b/ingestion_server/ingestion_server/indexer_worker.py
@@ -112,6 +112,6 @@ def _self_destruct():
 )
 handler.setFormatter(formatter)
 root.addHandler(handler)
-api = falcon.API()
+api = falcon.App()
 api.add_route('/indexing_task', IndexingJobResource())
 api.add_route('/healthcheck', HealthcheckResource())


Test patch:
Repository: WordPress/openverse-api

Issue:
[Bug] API not adding CORS headers to responses
## Description
<!-- Concisely describe the bug. -->
When trying to use the API locally with the Frontend project I got the following message in the browser's console:

> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8000/v1/images?q=birds. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

This is from Firefox but a very similar one will show in Chrome too, preventing to get any useful response.

## Reproduction
<!-- Provide detailed steps to reproduce the bug. -->
1. Run this project normally (instructions in README)
2. Simultaneously, run the [Frontend][ov-fe] project using the local API
```bash
API_URL=http://localhost:8000/v1/ npm run dev
```
3. Open the browser and try to search anything
4. See error.


## Expectation
<!-- Concisely describe what you expected to happen. -->
It should be possible to access the API from any domain, as [_is already intended_][ov-api-config], but a slightly misconfiguration is impeding.


## Screenshots
<!-- Add screenshots to show the problem; or delete the section entirely. -->

<img width="814" alt="Screen Shot 2021-08-25 at 7 17 32 PM" src="https://user-images.githubusercontent.com/9145885/130876641-1e2f13ac-2717-4ce1-a555-46aa94133967.png">


[ov-fe]: https://github.com/WordPress/openverse-frontend
[ov-api-config]: https://github.com/WordPress/openverse-api/blob/main/openverse-api/catalog/settings.py#L286:L287

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [x] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/openverse-api/catalog/settings.py b/openverse-api/catalog/settings.py
--- a/openverse-api/catalog/settings.py
+++ b/openverse-api/catalog/settings.py
@@ -77,12 +77,12 @@
 MIDDLEWARE = [
     'django.middleware.security.SecurityMiddleware',
     'django.contrib.sessions.middleware.SessionMiddleware',
+    'corsheaders.middleware.CorsMiddleware',
     'django.middleware.common.CommonMiddleware',
     'django.middleware.csrf.CsrfViewMiddleware',
     'django.contrib.auth.middleware.AuthenticationMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware',
     'django.middleware.clickjacking.XFrameOptionsMiddleware',
-    'corsheaders.middleware.CorsMiddleware',
     'oauth2_provider.middleware.OAuth2TokenMiddleware'
 ]
 


Test patch:
Repository: WordPress/openverse-api

Issue:
Dead link filtering reduces number of queryable pages 
## Description
<!-- Concisely describe the bug. -->
The dead link filtering algorithm significantly limits the pages that can be retrieved.

## Reproduction

0. Go to audio search endpoint: `http://localhost:8000/v1/audio/`. Note the number of pages.
1. Go to a page number that's quite high but less than 250, like 200: `http://localhost:8000/v1/audio/?page=200`
2. See the deep pagination error. This should not be happening.

## Expectation
<!-- Concisely describe what you expected to happen. -->
Page count calculation should be correct and should allow pagination upto the stated limit.

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.


Discussion:

Resolving patch:
diff --git a/openverse-api/catalog/api/controllers/search_controller.py b/openverse-api/catalog/api/controllers/search_controller.py
--- a/openverse-api/catalog/api/controllers/search_controller.py
+++ b/openverse-api/catalog/api/controllers/search_controller.py
@@ -485,6 +485,8 @@ def _get_result_and_page_count(response_obj: Response, results: List[Hit],
     """
     result_count = response_obj.hits.total.value
     natural_page_count = int(result_count / page_size)
+    if natural_page_count % page_size != 0:
+        natural_page_count += 1
     last_allowed_page = int((5000 + page_size / 2) / page_size)
     page_count = min(natural_page_count, last_allowed_page)
     if len(results) < page_size and page_count == 0:


Test patch:
Repository: WordPress/openverse-api

Issue:
[Bug] The integration tests are flaky sometimes
## Description
<!-- Concisely describe the bug. -->
The integration tests used in the CI workflow are sometimes flaky. In many cases, it's the fault of the `load_sample_data.sh` script:
- Uses arbitaray 2s and 30s intervals between ingestion steps, which is sometimes not enough. A better solution would be to [check the task status](https://github.com/WordPress/openverse-api/blob/34a089ed9b688624ff7855e0c63039913b274012/ingestion_server/ingestion_server/api.py#L180) till it completes successfully.
- Is not 100% idempotent so re-running it can have different outcomes. At the very least, the script should ensure that it cleans the slate before writing on it.

It also does not help that `workflows/integration-tests.yml` does not raise an error when any step fails, it just retries till it times out.

The goal is to reduce the flakiness using proper monitoring for the completion of the previous step and error messages in case of failures.

## Reproduction
<!-- Provide detailed steps to reproduce the bug. -->
Being flaky is by definition, hard to reproduce. Just running the CI several times should lead to the occurrence of such events.

## Expectation
<!-- Concisely describe what you expected to happen. -->
Tests are very important for confidence and so should not be flaky.

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/ingestion_server/ingestion_server/api.py b/ingestion_server/ingestion_server/api.py
--- a/ingestion_server/ingestion_server/api.py
+++ b/ingestion_server/ingestion_server/api.py
@@ -26,6 +26,12 @@
 SINCE_DATE = "since_date"
 
 
+class Health:
+    def on_get(self, req, resp):
+        resp.status = falcon.HTTP_200
+        resp.media = {"status": "200 OK"}
+
+
 class TaskResource:
     def __init__(self, tracker: TaskTracker):
         self.tracker = tracker
@@ -176,6 +182,7 @@ def create_api(log=True):
     task_tracker = TaskTracker()
     task_resource = TaskResource(task_tracker)
     get_task_status = TaskStatus(task_tracker)
+    _api.add_route("/", Health())
     _api.add_route("/task", task_resource)
     _api.add_route("/task/{task_id}", get_task_status)
     _api.add_route("/worker_finished", WorkerFinishedResource())
diff --git a/openverse_api/catalog/api/views/site_views.py b/openverse_api/catalog/api/views/site_views.py
--- a/openverse_api/catalog/api/views/site_views.py
+++ b/openverse_api/catalog/api/views/site_views.py
@@ -42,7 +42,7 @@ class HealthCheck(APIView):
     swagger_schema = None
 
     def get(self, request, format=None):
-        return Response("", status=200)
+        return Response({"status": "200 OK"}, status=200)
 
 
 class Register(APIView):


Test patch:
Repository: WordPress/openverse-api

Issue:
page_size for /related/ endpoint is set incorrectly and causes 400 error
Currently on the main branch, `/related/` endpoint returns a 400 error. This is caused by incorrectly setting the `page_size` to the number of search results for related. When setting the `page_size`, we also check for the number to be between 0 and 500, and when the number of search results is more than that, the `/related` endpoint returns a 400 error:

https://github.com/WordPress/openverse-api/blob/13673ee8c196c9a6b786a1322d6d65b97c6ca708/openverse_api/catalog/api/views/media_views.py#L97-L108

This is the code that throws the error, in `openverse_api/catalog/api/utils/pagination.py`:
```
 @page_size.setter
    def page_size(self, value):
        logging.info(f'Setting page size to {value}')
        if value is None:
            return
        value = int(value)  # convert str params to int
        if value <= 0 or value > 500:
            raise get_api_exception("Page size must be between 0 & 500.", 400)
        self._page_size = value
```


Discussion:

Resolving patch:
diff --git a/openverse_api/catalog/api/views/media_views.py b/openverse_api/catalog/api/views/media_views.py
--- a/openverse_api/catalog/api/views/media_views.py
+++ b/openverse_api/catalog/api/views/media_views.py
@@ -105,7 +105,8 @@ def related(self, request, identifier=None, *_, **__):
             )
             self.paginator.result_count = num_results
             self.paginator.page_count = 1
-            self.paginator.page_size = num_results
+            # `page_size` refers to the maximum number of related images to return.
+            self.paginator.page_size = 10
         except ValueError as e:
             raise get_api_exception(getattr(e, "message", str(e)))
 


Test patch:
Repository: WordPress/openverse-api

Issue:
Add Sentry to API
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
We don't have any visibility into the API service. Sentry would be a good and easy first step.

## Description
<!-- Describe the feature and how it solves the problem. -->
Let's add Sentry. Long term we have goals of adding other monitoring but Sentry is a good and easy first step.

## Additional context
<!-- Add any other context about the feature here; or delete the section entirely. -->

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [x] 🙋 I would be interested in implementing this feature.



Discussion:

Resolving patch:
diff --git a/openverse_api/catalog/settings.py b/openverse_api/catalog/settings.py
--- a/openverse_api/catalog/settings.py
+++ b/openverse_api/catalog/settings.py
@@ -13,7 +13,9 @@
 from pathlib import Path
 from socket import gethostbyname, gethostname
 
+import sentry_sdk
 from decouple import config
+from sentry_sdk.integrations.django import DjangoIntegration
 
 
 # Build paths inside the project like this: BASE_DIR.join('dir', 'subdir'...)
@@ -35,6 +37,8 @@
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = config("DJANGO_DEBUG_ENABLED", default=False, cast=bool)
 
+PYTHON_ENV = config("PYTHON_ENV", default="production")
+
 ALLOWED_HOSTS = [
     "api-dev.openverse.engineering",
     "api.openverse.engineering",
@@ -322,3 +326,18 @@
 
 # Whether to boost results by authority and popularity
 USE_RANK_FEATURES = config("USE_RANK_FEATURES", default=True, cast=bool)
+
+SENTRY_DSN = config(
+    "SENTRY_DSN",
+    default="https://08f4706d16004f57bcd37eb907bfc2e7@o787041.ingest.sentry.io/6107216",
+)
+SENTRY_SAMPLE_RATE = config("SENTRY_SAMPLE_RATE", default=1.0, cast=float)
+
+if not DEBUG:
+    sentry_sdk.init(
+        dsn=SENTRY_DSN,
+        integrations=[DjangoIntegration()],
+        traces_sample_rate=SENTRY_SAMPLE_RATE,
+        send_default_pii=False,
+        environment=PYTHON_ENV,
+    )


Test patch:
Repository: WordPress/openverse-api

Issue:
Replace PYTHON_ENV with ENVIRONMENT
## Description
<!-- Concisely describe the bug. -->
Our deployment infrastructure already configures an `ENVIRONMENT` variable and it is already used in the ingestion server code. Rather than introduce a new one that duplicates that, we can just use that everywhere.

Making this change will solve the issue we're seeing Sentry where staging is reporting itself as `production`.

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [x] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -39,7 +39,7 @@
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = config("DJANGO_DEBUG_ENABLED", default=False, cast=bool)
 
-PYTHON_ENV = config("PYTHON_ENV", default="production")
+ENVIRONMENT = config("ENVIRONMENT", default="local")
 
 ALLOWED_HOSTS = [
     "api-dev.openverse.engineering",
@@ -318,5 +318,5 @@
         integrations=[DjangoIntegration()],
         traces_sample_rate=SENTRY_SAMPLE_RATE,
         send_default_pii=False,
-        environment=PYTHON_ENV,
+        environment=ENVIRONMENT,
     )


Test patch:
Repository: WordPress/openverse-api

Issue:
Sound category mismatch
## Description
<!-- Concisely describe the bug. -->
The `sound` category for audio doesn't work on the front-end.
There seems to be a mismatch between the `audio` category of `sound_effect`:
If you go to `https://api.openverse.engineering/v1/audio/?q=cat&categories=sound`, you will get a 400 response:
```
HTTP 400 Bad Request
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "detail": {
        "categories": [
            "Invalid category: sound. Available options: {'music', 'audiobook', 'podcast', 'news', 'sound_effect'}"
        ]
    }
}
```

However, if you access a single audio result, you will see that it returns `sound` for the category:
https://api.openverse.engineering/v1/audio/1bb94f50-009c-4371-a605-dd289562a9f5/

## Expectation
<!-- Concisely describe what you expected to happen. -->
Both the query category parameter and the result category property for sound effect should have the same name.

## Additional context
The catalog sets the category as `sound`, so that is the value we get from the database:
https://github.com/WordPress/openverse-catalog/blob/cb19f839e96de7ae1a55e8b7dc82a7d2bf5588e8/openverse_catalog/dags/providers/provider_api_scripts/freesound.py#L33-L34

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.



Discussion:

Resolving patch:
diff --git a/api/catalog/api/serializers/audio_serializers.py b/api/catalog/api/serializers/audio_serializers.py
--- a/api/catalog/api/serializers/audio_serializers.py
+++ b/api/catalog/api/serializers/audio_serializers.py
@@ -88,6 +88,7 @@ def validate_categories(value):
             "podcast",
             "news",
             "audiobook",
+            "pronunciation",
         }
         _validate_enum("category", valid_categories, value)
         return value.lower()


Test patch:
Repository: WordPress/openverse-api

Issue:
Create a local SSL certificate with `mkcert`
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->

Currently if one wants to use `https://` locally, they have to bypass browser warnings about the page being insecure or implement a completely custom solution for creating certificates. Or, perhaps worst, not use SSL at all. It's worth noting that local SSL isn't about _security_, rather parity with production for testing. See this article for a list of [when to use SSL locally](https://web.dev/when-to-use-local-https/).

## Description
<!-- Describe the feature and how it solves the problem. -->

Similar to the implementation in the [frontend](https://github.com/WordPress/openverse-frontend/pull/1144), we should use the [mkcert](https://github.com/FiloSottile/mkcert#installation) library to enable contributors to set up a local certificate authority and create a certificate for local development. Some thing we'd likely want to do in this PR:

- Add instructions for installing and configuring mkcert
- Add a just script for generating the certificate
- Add a mechanism to tell Django about the local certificate and key
- Ignore the local certificate and key in `.gitignore`


Discussion:

Resolving patch:
diff --git a/api/catalog/api/serializers/base.py b/api/catalog/api/serializers/base.py
--- a/api/catalog/api/serializers/base.py
+++ b/api/catalog/api/serializers/base.py
@@ -16,4 +16,9 @@ def __init__(self, scheme=settings.API_LINK_SCHEME, *args, **kwargs):
 
     def get_url(self, *args, **kwargs):
         url = super().get_url(*args, **kwargs)
-        return re.sub(r"^\w+://", f"{self.scheme}://", url, 1)
+
+        # Only rewrite URLs if a fixed scheme is provided
+        if self.scheme is not None:
+            re.sub(r"^\w+://", f"{self.scheme}://", url, 1)
+
+        return url
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -53,6 +53,7 @@
 
 if DEBUG:
     ALLOWED_HOSTS += [
+        "dev.openverse.test",  # used in local development
         "localhost",
         "127.0.0.1",
         "0.0.0.0",
@@ -307,7 +308,14 @@
 USE_RANK_FEATURES = config("USE_RANK_FEATURES", default=True, cast=bool)
 
 # The scheme to use for the hyperlinks in the API responses
-API_LINK_SCHEME = config("API_LINK_SCHEME", default="https")
+API_LINK_SCHEME = config("API_LINK_SCHEME", default=None)
+
+# Proxy handling, for production
+if config("IS_PROXIED", default=True, cast=bool):
+    # https://docs.djangoproject.com/en/4.0/ref/settings/#use-x-forwarded-host
+    USE_X_FORWARDED_HOST = True
+    # https://docs.djangoproject.com/en/4.0/ref/settings/#secure-proxy-ssl-header
+    SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
 
 SENTRY_DSN = config(
     "SENTRY_DSN",


Test patch: