internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mozilla/pontoon

Issue:
User with team manager role and Admin rights appear as Contributor in the locale contributors view
When viewing contributions to a locale and its associated projects, users with a Pontoon admin role in addition to a team manager role will only appear as a Contributor for that locale.

Example:
https://pontoon.mozilla.org/sl/contributors/?period=12

Current: Matjaž Horvat appears under contributors for Slovenian locale
Expected: Matjaž Horvat appears under managers Slovenian locale

Note: Under their user profile their Team manager role will be correctly displayed.
https://pontoon.mozilla.org/contributors/mathjazz/


Discussion:

Resolving patch:
diff --git a/pontoon/base/models.py b/pontoon/base/models.py
--- a/pontoon/base/models.py
+++ b/pontoon/base/models.py
@@ -251,14 +251,14 @@ def user_role(self, managers=None, translators=None):
 
 
 def user_locale_role(self, locale):
-    if self.is_superuser:
-        return "Admin"
-    if self.pk is None or self.profile.system_user:
-        return "System User"
     if self in locale.managers_group.user_set.all():
         return "Manager"
     if self in locale.translators_group.user_set.all():
         return "Translator"
+    if self.is_superuser:
+        return "Admin"
+    if self.pk is None or self.profile.system_user:
+        return "System User"
     else:
         return "Contributor"
 


Test patch:
diff --git a/pontoon/base/tests/models/test_user.py b/pontoon/base/tests/models/test_user.py
--- a/pontoon/base/tests/models/test_user.py
+++ b/pontoon/base/tests/models/test_user.py
@@ -56,3 +56,7 @@ def test_user_locale_role(user_a, user_b, user_c, locale_a):
     # Manager
     locale_a.managers_group.user_set.add(user_c)
     assert user_c.locale_role(locale_a) == "Manager"
+
+    # Admin and Manager
+    locale_a.managers_group.user_set.add(user_a)
+    assert user_a.locale_role(locale_a) == "Manager"
Repository: mozilla/pontoon

Issue:
Project+locale AJAX endpoint logic is not uniform
At the moment, `/:code/:slug/ajax/` and `/:code/:slug/ajax/tags/` check if the locale & project exist and are visible to the user, while `/:code/:slug/ajax/contributors/` and `/:code/:slug/ajax/insights/` check that the ProjectLocale exists. [All of these](https://github.com/mozilla/pontoon/blob/master/pontoon/localizations/views.py) should check both conditions.


Discussion:

Resolving patch:
diff --git a/pontoon/localizations/views.py b/pontoon/localizations/views.py
--- a/pontoon/localizations/views.py
+++ b/pontoon/localizations/views.py
@@ -71,6 +71,9 @@ def ajax_resources(request, code, slug):
         slug=slug,
     )
 
+    # Check if ProjectLocale exists
+    get_object_or_404(ProjectLocale, locale=locale, project=project)
+
     # Amend the parts dict with latest activity info.
     translatedresources_qs = TranslatedResource.objects.filter(
         resource__project=project, locale=locale
@@ -150,6 +153,9 @@ def ajax_tags(request, code, slug):
     locale = get_object_or_404(Locale, code=code)
     project = get_object_or_404(Project.objects.visible_for(request.user), slug=slug)
 
+    # Check if ProjectLocale exists
+    get_object_or_404(ProjectLocale, locale=locale, project=project)
+
     if not project.tags_enabled:
         raise Http404
 
@@ -174,6 +180,8 @@ def ajax_insights(request, code, slug):
     if not settings.ENABLE_INSIGHTS:
         raise ImproperlyConfigured("ENABLE_INSIGHTS variable not set in settings.")
 
+    get_object_or_404(Locale, code=code)
+    get_object_or_404(Project.objects.visible_for(request.user), slug=slug)
     pl = get_object_or_404(ProjectLocale, locale__code=code, project__slug=slug)
     insights = get_insights(locale=pl.locale, project=pl.project)
 
@@ -188,6 +196,10 @@ class LocalizationContributorsView(ContributorsMixin, DetailView):
     template_name = "localizations/includes/contributors.html"
 
     def get_object(self):
+        get_object_or_404(Locale, code=self.kwargs["code"])
+        get_object_or_404(
+            Project.objects.visible_for(self.request.user), slug=self.kwargs["slug"]
+        )
         return get_object_or_404(
             ProjectLocale,
             locale__code=self.kwargs["code"],


Test patch:
diff --git a/pontoon/localizations/tests/test_views.py b/pontoon/localizations/tests/test_views.py
--- a/pontoon/localizations/tests/test_views.py
+++ b/pontoon/localizations/tests/test_views.py
@@ -10,6 +10,7 @@
     ResourceFactory,
     TranslationFactory,
     TranslatedResourceFactory,
+    ProjectLocaleFactory,
 )
 
 
@@ -18,6 +19,7 @@
 @patch.object(Locale, "parts_stats")
 def test_latest_activity(mock_parts_stats, mock_render, client, project_a, locale_a):
     """Ensure that the latest_activity field is added to parts."""
+    ProjectLocaleFactory.create(locale=locale_a, project=project_a)
     resource = ResourceFactory.create(project=project_a, path="has/stats.po")
     resource2 = ResourceFactory.create(project=project_a, path="has/stats2.po")
Repository: mozilla/pontoon

Issue:
Document DDoS mitigation
The `BLOCKED_IPS` env variable is not documented here:
https://mozilla-pontoon.readthedocs.io/en/latest/admin/deployment.html.

We should also add a paragraph here on DDoS mitigation:
https://mozilla-pontoon.readthedocs.io/en/latest/admin/maintenance.html


Discussion:

Resolving patch:
diff --git a/pontoon/settings/base.py b/pontoon/settings/base.py
--- a/pontoon/settings/base.py
+++ b/pontoon/settings/base.py
@@ -267,6 +267,7 @@ def _default_from_email():
     "django_ace",
 )
 
+# A list of IP addresses to be blocked from accessing the app, because they are DDoS'ing the server
 BLOCKED_IPS = os.environ.get("BLOCKED_IPS", "").split(",")
 
 MIDDLEWARE = (


Test patch:
Repository: mozilla/pontoon

Issue:
Integrate debug_sql()
Integrate https://github.com/mozilla-l10n/pontoon-scripts/blob/main/dev/debug_sql_performance.py into Pontoon for easier use.


Discussion:

Resolving patch:
diff --git a/pontoon/dev/__init__.py b/pontoon/dev/__init__.py
new file mode 100644
diff --git a/pontoon/dev/utils.py b/pontoon/dev/utils.py
new file mode 100644
--- /dev/null
+++ b/pontoon/dev/utils.py
@@ -0,0 +1,34 @@
+"""
+Author: phlax
+
+Usage:
+with debug_sql():
+    code_with_some_db_action()
+"""
+
+import logging
+from contextlib import contextmanager
+from django.db import connection
+
+
+log = logging.getLogger(__name__)
+
+
+def log_new_queries(queries):
+    new_queries = list(connection.queries[queries:])
+
+    for query in new_queries:
+        log.debug(query["time"])
+        log.debug("\t%s", query["sql"])
+
+    log.debug("total db calls: %s", len(new_queries))
+
+
+@contextmanager
+def debug_sql():
+    queries = len(connection.queries)
+
+    try:
+        yield
+    finally:
+        log_new_queries(queries)


Test patch:
Repository: mozilla/pontoon

Issue:
Hitting a server error when accessing a Tag page of a Tag without any resoures associated to it
This is a regression from https://github.com/mozilla/pontoon/commit/1dcd7382221f7b943b9b743ee32322f7233f6a86.


Discussion:

Resolving patch:
diff --git a/pontoon/tags/utils.py b/pontoon/tags/utils.py
--- a/pontoon/tags/utils.py
+++ b/pontoon/tags/utils.py
@@ -13,7 +13,9 @@ def __init__(self, **kwargs):
         self.project = kwargs.get("project")
         self.locale = kwargs.get("locale")
         self.slug = kwargs.get("slug")
-        self.tag = Tag.objects.filter(project=self.project, slug=self.slug).first()
+        self.tag = Tag.objects.filter(
+            project=self.project, slug=self.slug, resources__isnull=False
+        ).first()
 
     def get(self):
         tags = (


Test patch:
diff --git a/pontoon/tags/tests/test_views.py b/pontoon/tags/tests/test_views.py
--- a/pontoon/tags/tests/test_views.py
+++ b/pontoon/tags/tests/test_views.py
@@ -3,23 +3,37 @@
 from django.urls import reverse
 
 from pontoon.base.models import Priority
+from pontoon.test.factories import TagFactory, TranslatedResourceFactory
 
 
 @pytest.mark.django_db
-def test_view_project_tag_locales(client, project_a, tag_a):
+def test_view_project_tag_locales(client, locale_a, project_a, resource_a):
+    TranslatedResourceFactory.create(resource=resource_a, locale=locale_a)
+
+    tag = TagFactory.create(
+        slug="tag",
+        name="Tag",
+    )
+
     url = reverse(
         "pontoon.tags.project.tag",
-        kwargs=dict(project=project_a.slug, tag=tag_a.slug),
+        kwargs=dict(project=project_a.slug, tag=tag.slug),
     )
 
-    # tag is not associated with project
-    project_a.tag_set.remove(tag_a)
+    # tag is not associated with a project
     response = client.get(url)
     assert response.status_code == 404
 
-    project_a.tag_set.add(tag_a)
-    tag_a.priority = Priority.NORMAL
-    tag_a.save()
+    tag.project = project_a
+    tag.save()
+
+    # tag is not associated with a resource
+    response = client.get(url)
+    assert response.status_code == 404
+
+    tag.resources.add(resource_a)
+    tag.save()
+
     response = client.get(url)
     assert response.status_code == 200
 
@@ -31,7 +45,7 @@ def test_view_project_tag_locales(client, project_a, tag_a):
     assert response.context_data["project"] == project_a
 
     res_tag = response.context_data["tag"]
-    assert res_tag == tag_a
+    assert res_tag == tag
 
 
 @pytest.mark.django_db
Repository: mozilla/pontoon

Issue:
Speed up Project Notifications tab
The Notifications tab takes a while to load if multiple manual notifications have been sent to project contributors.


Discussion:

Resolving patch:
diff --git a/pontoon/projects/views.py b/pontoon/projects/views.py
--- a/pontoon/projects/views.py
+++ b/pontoon/projects/views.py
@@ -186,20 +186,27 @@ def ajax_notifications(request, slug):
                 identifier=identifier,
             )
 
-    # Detect previously sent notifications using a unique identifier
-    # TODO: We should simplify this with a custom Notifications model
-    notifications_map = {}
-
-    for notification in Notification.objects.filter(
-        description__isnull=False,
-        target_content_type=ContentType.objects.get_for_model(project),
-        target_object_id=project.id,
-    ):
-        identifier = notification.data["identifier"]
-        if identifier not in notifications_map:
-            notifications_map[identifier] = notification
-
-    notifications = list(notifications_map.values())
+    notifications = list(
+        Notification.objects.filter(
+            description__isnull=False,
+            target_content_type=ContentType.objects.get_for_model(project),
+            target_object_id=project.id,
+        )
+        # Each project notification is stored in one Notification instance per user. To
+        # identify unique project Notifications, we use the identifier stored in the
+        # Notification.data field.
+        #
+        # PostgreSQL allows us to retrieve Notifications with unique Notification.data
+        # fields by combining .order_by(*fields) and .distinct(*fields) calls. Read more:
+        # https://docs.djangoproject.com/en/3.2/ref/models/querysets/#distinct
+        #
+        # That approach doesn't allow us to order Notifications by their timestamp, so
+        # we have to do that in python below.
+        .order_by("data")
+        .distinct("data")
+        .prefetch_related("actor", "target")
+    )
+
     notifications.sort(key=lambda x: x.timestamp, reverse=True)
 
     # Recipient shortcuts


Test patch:
Repository: mozilla/pontoon

Issue:
Translation Memory requests take longer than Machine Translation requests
We should bring (internal) translation memory request times at least on par with (external) machine translation request times.

Possible action items:
- Consider using indexes in the TranslationMemoryEntry table.
- Consider normalizing additional fields in the TranslationMemoryEntry table.
- [Cache TM requests](https://github.com/mozilla/pontoon/issues/3077).


Discussion:

Resolving patch:
diff --git a/pontoon/base/migrations/0056_remove_fuzzy_and_non-approved_tm_entries.py b/pontoon/base/migrations/0056_remove_fuzzy_and_non-approved_tm_entries.py
new file mode 100644
--- /dev/null
+++ b/pontoon/base/migrations/0056_remove_fuzzy_and_non-approved_tm_entries.py
@@ -0,0 +1,25 @@
+# Generated by Django 3.2.24 on 2024-03-28 10:39
+
+from django.db import migrations
+
+
+def remove_unused_tm_entries(apps, schema_editor):
+    """Remove TM entries that belong to fuzzy Translations and Translations that are not approved."""
+    TranslationMemoryEntry = apps.get_model("base", "TranslationMemoryEntry")
+
+    TranslationMemoryEntry.objects.filter(translation__fuzzy=True).delete()
+    TranslationMemoryEntry.objects.filter(translation__approved=False).delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("base", "0055_populate_resource_order"),
+    ]
+
+    operations = [
+        migrations.RunPython(
+            code=remove_unused_tm_entries,
+            reverse_code=migrations.RunPython.noop,
+        ),
+    ]
diff --git a/pontoon/machinery/utils.py b/pontoon/machinery/utils.py
--- a/pontoon/machinery/utils.py
+++ b/pontoon/machinery/utils.py
@@ -263,11 +263,9 @@ def sort_by_quality(entity):
 
 
 def get_translation_memory_data(text, locale, pk=None):
-    entries = (
-        base.models.TranslationMemoryEntry.objects.filter(locale=locale)
-        .minimum_levenshtein_ratio(text)
-        .exclude(translation__approved=False, translation__fuzzy=False)
-    )
+    entries = base.models.TranslationMemoryEntry.objects.filter(
+        locale=locale
+    ).minimum_levenshtein_ratio(text)
 
     # Exclude existing entity
     if pk:


Test patch:
Repository: mozilla/pontoon

Issue:
Collation specialization appears to be out of date
We currently build our own [postgres docker image](https://github.com/mozilla/pontoon/blob/main/docker/postgres/Dockerfile) and do [extra steps during test setup](https://github.com/mozilla/pontoon/blob/main/pontoon/test/setup.py) to help ensure that we have a `tr_tr` collation available in Postgres. This was added to resolve [this bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1346180) in 2017.

Reviewing the situation now, it looks like the same behaviour is available "out of the box" with our current Django and Postgres versions by using the `tr-x-icu` collation. We should despecialize what we're doing and use that (and other ICU-backed locales) where appropriate.


Discussion:

Resolving patch:
diff --git a/pontoon/base/migrations/0058_remove_tr_tr_collation.py b/pontoon/base/migrations/0058_remove_tr_tr_collation.py
new file mode 100644
--- /dev/null
+++ b/pontoon/base/migrations/0058_remove_tr_tr_collation.py
@@ -0,0 +1,20 @@
+# Generated manually on 2024-04-04
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("base", "0057_remove_lang_format"),
+    ]
+
+    operations = [
+        migrations.RunSQL(
+            sql="UPDATE base_locale SET db_collation='tr-x-icu' WHERE db_collation='tr_tr';",
+        ),
+        migrations.RunSQL(
+            sql="DROP COLLATION IF EXISTS tr_tr;",
+            reverse_sql="CREATE COLLATION tr_tr (LOCALE = 'tr_TR.utf8');",
+        ),
+    ]


Test patch:
Repository: mozilla/pontoon

Issue:
Update to Google Analytics 4
We should update to Google Analytics 4 or our property will be permanently deleted on July 1, 2024.

Until then our Universal Analytics property remains unchanged and continues to collect data.


Discussion:

Resolving patch:
diff --git a/pontoon/settings/base.py b/pontoon/settings/base.py
--- a/pontoon/settings/base.py
+++ b/pontoon/settings/base.py
@@ -825,8 +825,8 @@ def _allowed_hosts():
     "'unsafe-eval'",
     "'sha256-fDsgbzHC0sNuBdM4W91nXVccgFLwIDkl197QEca/Cl4='",
     # Rules related to Google Analytics
-    "'sha256-G5/M3dBlZdlvno5Cibw42fbeLr2PTEGd1M909Z7vPZE='",
-    "https://www.google-analytics.com/analytics.js",
+    "'sha256-MAn2iEyXLmB7sfv/20ImVRdQs8NCZ0A5SShdZsZdv20='",
+    "https://www.googletagmanager.com/gtag/js",
 )
 CSP_STYLE_SRC = (
     "'self'",


Test patch:
Repository: mozilla/pontoon

Issue:
Upgrade Chart.js
We use an outdated Chart.js library. Let's upgrade it to latest version.


Discussion:

Resolving patch:
diff --git a/pontoon/settings/base.py b/pontoon/settings/base.py
--- a/pontoon/settings/base.py
+++ b/pontoon/settings/base.py
@@ -544,7 +544,8 @@ def _default_from_email():
     },
     "insights": {
         "source_filenames": (
-            "js/lib/Chart.bundle.js",
+            "js/lib/chart.umd.min.js",
+            "js/lib/chartjs-adapter-date-fns.bundle.min.js",
             "js/insights_charts.js",
             "js/insights.js",
         ),
@@ -552,7 +553,8 @@ def _default_from_email():
     },
     "localization": {
         "source_filenames": (
-            "js/lib/Chart.bundle.js",
+            "js/lib/chart.umd.min.js",
+            "js/lib/chartjs-adapter-date-fns.bundle.min.js",
             "js/table.js",
             "js/progress-chart.js",
             "js/tabs.js",
@@ -564,7 +566,8 @@ def _default_from_email():
     },
     "project": {
         "source_filenames": (
-            "js/lib/Chart.bundle.js",
+            "js/lib/chart.umd.min.js",
+            "js/lib/chartjs-adapter-date-fns.bundle.min.js",
             "js/table.js",
             "js/request.js",
             "js/progress-chart.js",
@@ -586,7 +589,8 @@ def _default_from_email():
     },
     "team": {
         "source_filenames": (
-            "js/lib/Chart.bundle.js",
+            "js/lib/chart.umd.min.js",
+            "js/lib/chartjs-adapter-date-fns.bundle.min.js",
             "js/table.js",
             "js/progress-chart.js",
             "js/double_list_selector.js",
@@ -610,7 +614,8 @@ def _default_from_email():
     },
     "profile": {
         "source_filenames": (
-            "js/lib/Chart.bundle.js",
+            "js/lib/chart.umd.min.js",
+            "js/lib/chartjs-adapter-date-fns.bundle.min.js",
             "js/insights_charts.js",
             "js/profile.js",
         ),


Test patch:
Repository: mozilla/pontoon

Issue:
Add ability to mention users using their username
As part of the redesigned profile page, we've introduced usernames.

We should make use of them in user mentions in comments.


Discussion:

Resolving patch:
diff --git a/pontoon/base/views.py b/pontoon/base/views.py
--- a/pontoon/base/views.py
+++ b/pontoon/base/views.py
@@ -676,6 +676,8 @@ def get_users(request):
         .exclude(profile__system_user=True)
         # Exclude deleted users
         .exclude(email__regex=r"^deleted-user-(\w+)@example.com$")
+        # Prefetch profile for retrieving username
+        .prefetch_related("profile")
     )
     payload = []
 
@@ -685,6 +687,7 @@ def get_users(request):
                 "gravatar": u.gravatar_url(44),
                 "name": u.name_or_email,
                 "url": u.profile_url,
+                "username": u.profile.username,
             }
         )
 


Test patch:
Repository: conda/conda-build

Issue:
Conda Build should check that the meta.yaml is valid
The `info/recipe/meta.yaml` file in a conda package may be invalid. Conda should check this before returning a successful build result. 

From conda/conda-recipes#98



Discussion:
I don't see how it worked at all, since conda gets the metadata from meta.yaml. 

@asmeurer @ilanschnell this is definitely still an issue. 

This occurs for every build using jinja templates. and I imagine it would happen for selectors as well.

I would have sent a PR but I can not find the code where the meta.yaml is copied.

I would not copy the file directly but do a `yaml.dump` on the [`.meta` attribute of a MetaData object](https://github.com/conda/conda-build/blob/73dc3244589b1838d6d2b483b73de8ee678d667c/conda_build/metadata.py#L183)

This is somewhat of a high priority and is blocking me on wakari development


Resolving patch:
diff --git a/conda_build/build.py b/conda_build/build.py
--- a/conda_build/build.py
+++ b/conda_build/build.py
@@ -15,6 +15,7 @@
 import tarfile
 from io import open
 from os.path import exists, isdir, isfile, islink, join
+import yaml
 
 # Python 2.x backward compatibility
 if sys.version_info < (3, 0):
@@ -143,6 +144,12 @@ def create_info_files(m, files, include_recipe=True):
             else:
                 shutil.copy(src_path, dst_path)
 
+    if isfile(join(recipe_dir, 'meta.yaml')):
+        shutil.move(join(recipe_dir, 'meta.yaml'), join(recipe_dir, 'meta.yaml.orig'))
+
+    with open(join(recipe_dir, 'meta.yaml'), 'w', encoding='utf-8') as fo:
+        yaml.safe_dump(m.meta, fo)
+
     with open(join(info_dir, 'files'), 'w', encoding='utf-8') as fo:
         for f in files:
             if sys.platform == 'win32':


Test patch:
Repository: conda/conda-build

Issue:
Conda should automatically test imports of py_modules as well as packages
Should be pretty simple to add.

I'd make a PR myself but I haven't been able to make a dev install for conda-build yet (#141).

More context: https://github.com/conda/conda-build/issues/139#issuecomment-49466828



Discussion:
That issue #141 is fixed, but the conda_build setup.py does not handle installing bdist_conda. You'll need to symlink it to ~/anaconda/lib/python-2.7/distutils/commands.

We should add this to conda skeleton pypi as well. 


Resolving patch:
diff --git a/bdist_conda.py b/bdist_conda.py
--- a/bdist_conda.py
+++ b/bdist_conda.py
@@ -232,11 +232,10 @@ def run(self):
             if 'setuptools' in d['requirements']['run']:
                 d['build']['preserve_egg_dir'] = True
 
-
             if metadata.conda_import_tests:
                 if metadata.conda_import_tests is True:
-                    if self.distribution.packages:
-                        d['test']['imports'] = self.distribution.packages
+                    d['test']['imports'] = ((self.distribution.packages or [])
+                                            + (self.distribution.py_modules or []))
                 else:
                     d['test']['imports'] = metadata.conda_import_tests
 


Test patch:
Repository: conda/conda-build

Issue:
Unable to build source.git_url based build recipes
See for an example: https://binstar.org/conda/conda-env/builds/2/3



Discussion:
cc @asmeurer @srossross 

I'm not sure what is going on there, and I can't reproduce it. Is it possible to add some debugging code in binstar build?

Also what version of git is used? That seems to matter a lot for these kinds of things. I wasn't able to reproduce it with 1.9.4.mysisgit.0 on 32-bit Windows 7.

`1.9.4.mysisgit.1` on windows server 2012

FYI, I was able to build https://github.com/conda/conda-env/tree/0c3e6563e56e549d77496604aeb0403e73e0326c in a Windows XP 32-bit system and got the expected results.  This appears to be something specific to the way binstar-build is invoking conda build.


Resolving patch:
diff --git a/conda_build/source.py b/conda_build/source.py
--- a/conda_build/source.py
+++ b/conda_build/source.py
@@ -83,7 +83,8 @@ def git_source(meta, recipe_dir):
     if git_url.startswith('.'):
         # It's a relative path from the conda recipe
         os.chdir(recipe_dir)
-        git_dn = abspath(expanduser(git_url)).replace('/', '_')
+        git_dn = abspath(expanduser(git_url))
+        git_dn = "_".join(git_dn.split(os.path.sep)[1:])
     else:
         git_dn = git_url.split(':')[-1].replace('/', '_')
     cache_repo = cache_repo_arg = join(GIT_CACHE, git_dn)


Test patch: