internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: translate/pootle

Issue:
Add test if TP uses a proper checker
This commit https://github.com/translate/pootle/commit/1d6ef1c987f2ee421b678fb9ac36e16175e4f364 fixed very hidden bug, let's add a test for it.



Discussion:

Resolving patch:
diff --git a/pytest_pootle/fixtures/models/translation_project.py b/pytest_pootle/fixtures/models/translation_project.py
--- a/pytest_pootle/fixtures/models/translation_project.py
+++ b/pytest_pootle/fixtures/models/translation_project.py
@@ -7,6 +7,8 @@
 # or later license. See the LICENSE file for a copy of the license and the
 # AUTHORS file for copyright and authorship information.
 
+import shutil
+
 import pytest
 
 
@@ -79,3 +81,25 @@ def afrikaans_vfolder_test(afrikaans, vfolder_test):
 def templates_tutorial(templates, tutorial):
     """Require Template Tutorial."""
     return _require_tp(templates, tutorial)
+
+
+def get_project_checkers():
+    from translate.filters import checks
+
+    return ['standard'] + list(checks.projectcheckers.keys())
+
+
+@pytest.fixture(params=get_project_checkers())
+def tp_checker_tests(request, english):
+    from pytest_pootle.factories import ProjectFactory
+
+    checker_name = request.param
+    project = ProjectFactory(
+        checkstyle=checker_name,
+        source_language=english)
+
+    def _remove_project_directory():
+        shutil.rmtree(project.get_real_path())
+    request.addfinalizer(_remove_project_directory)
+
+    return (checker_name, project)


Test patch:
diff --git a/tests/models/translationproject.py b/tests/models/translationproject.py
--- a/tests/models/translationproject.py
+++ b/tests/models/translationproject.py
@@ -116,3 +116,18 @@ def test_tp_stats_created_from_template(tutorial, fish, templates):
     assert stats['fuzzy'] == 0
     assert stats['suggestions'] == 0
     assert stats['critical'] == 0
+
+
+@pytest.mark.django_db
+def test_tp_checker(tp_checker_tests):
+    from translate.filters import checks
+
+    language = Language.objects.get(code="language0")
+    checker_name, project = tp_checker_tests
+    tp = TranslationProject.objects.create(project=project, language=language)
+
+    checkerclasses = [
+        checks.projectcheckers.get(tp.project.checkstyle,
+                                   checks.StandardChecker)
+    ]
+    assert [x.__class__ for x in tp.checker.checkers] == checkerclasses
Repository: translate/pootle

Issue:
calculate_checks requires running refresh_stats afterwards
If you run `calculate_checks` you will have to also run `refresh_stats` afterwards in order for the detected critical errors to be shown on the stats. This is necessary since if no errors are detected in the stats no links are properly rendered to directly see the errors in the editor.

Documentation says that `calculate_checks` **will flush existing caches and update the quality checks cache**, so probably the right fix is to align `calculate_checks` behavior with that statement.



Discussion:
@translate/core Other way to handle this is to remove that phrase from docs and instead warn about that it is necessary to run `refresh_stats`, and also adjust `calculate_checks` to output this message after being run.

as @dwaynebailey has mentioned - as calculate_checks makes the stats out-of-date - it makes sense that it should happen automatically.

However, my only concern is that a sysadmin/dev should also have the possibility to run only calculate_checks if they wish to - as both calculate_checks and refresh_stats are resource-intensive and take a while.

Having to run `refresh_stats` manually and documenting it as such is at best a band-aid or temporary fix.  

As an admin I'd be running `calculate_checks` because I need to update a changed check.  I'd expect that to mean that data on the front end gets updated.  Having to do it manually is not logical to me. Also, in a large project, you're updating a lot of data, most of which does not need updating.  While `calculate_checks` is in a better position to know exactly which data should be updated.

I've referenced this issue here https://github.com/translate/pootle/commit/16b607580795631eee96e9ba8043c2af42cd046e.

`refresh_stats` triggers jobs for all stores and updates all stats data.
`recalculate_checks` can trigger stats updating for checks data only and for those stores that are changed.
I think we need to fix `calculate_checks`. 


Resolving patch:
diff --git a/pootle/core/checks/checker.py b/pootle/core/checks/checker.py
--- a/pootle/core/checks/checker.py
+++ b/pootle/core/checks/checker.py
@@ -14,6 +14,7 @@
 from django.utils.functional import cached_property
 from django.utils.lru_cache import lru_cache
 
+from pootle.core.mixins.treeitem import CachedMethods
 from pootle_misc.checks import run_given_filters
 from pootle_misc.util import import_func
 from pootle_store.models import QualityCheck, Store, Unit
@@ -269,6 +270,7 @@ def update_store_caches(self, stores):
         """After completing QualityCheck updates expire caches for affected Stores.
         """
         for store in Store.objects.filter(pk__in=stores):
+            store.mark_dirty(CachedMethods.CHECKS, CachedMethods.MTIME)
             store.update_dirty_cache()
 
     def update_translated_unit(self, unit, checker=None):


Test patch:
Repository: translate/pootle

Issue:
Priority is displayed even if no virtual folders are being used
This is a regression: in a server with no virtual folders, units still display a default priority of _1.0_.



Discussion:
I don't think we should consider it a problem.

Would you mind elaborating? Because this is indeed a problem with no virtualfolders: there's no point in taking up UI space to display irrelevant and confusing information.

We denormalized priority to Unit to avoid having extra queries to check for VirtualFolders. What you suggest will require to check if virtualfolders app is enabled and VirtualFolderTreeItems exist for any Unit in the path being translated. I don't think it is such a big problem to display the priority if thus we avoid having additional queries.

`Unit.priority` has been around since the end of last year, and this is a recent regression. In fact, `git bisect` points to 508e4c19a, which dates back to 3 days ago.

``` python
priority = self.object.priority if 'virtualfolders' in settings.INSTALLED_APPS else None
```

is all what's needed. I'll PR the change.

great thanks @julen - you might need to change the test - not sure


Resolving patch:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -492,11 +492,15 @@ def get_sources(self):
         return sources
 
     def get_context_data(self, *args, **kwargs):
+        priority = (
+            self.object.priority if 'virtualfolder' in settings.INSTALLED_APPS
+            else None
+        )
         return {
             'unit': self.object,
             'form': self.get_unit_edit_form(),
             'comment_form': self.get_unit_comment_form(),
-            'priority': self.object.priority,
+            'priority': priority,
             'store': self.store,
             'directory': self.directory,
             'user': self.request.user,


Test patch:
Repository: translate/pootle

Issue:
TypeError: argument of type 'NoneType' is not iterable
In current master, the following exception can occur when users are working in the editor:

``` python
TypeError: argument of type 'NoneType' is not iterable
  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "pootle_misc/util.py", line 58, in wrapper
    return f(request, *args, **kwargs)
  File "pootle_store/views.py", line 218, in get_units
    and search_form.cleaned_data["uids"][0] not in uid_list)
```

Reproducible with:
`$ curl  -H "X-Requested-With: XMLHttpRequest" "https://server.tld/xhr/units/?path=%2Ffoo%2Fbar%2F&uids=123456"`



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/unit/search.py b/pootle/apps/pootle_store/unit/search.py
--- a/pootle/apps/pootle_store/unit/search.py
+++ b/pootle/apps/pootle_store/unit/search.py
@@ -145,7 +145,7 @@ def results(self):
         return self.sort_qs(self.filter_qs(self.units_qs))
 
     def search(self):
-        uid_list = None
+        uid_list = []
         begin = 0
         end = 2 * self.chunk_size
         if self.initial:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -214,7 +214,8 @@ def get_units(request):
         request.user, **search_form.cleaned_data).search()
 
     bad_uid = (
-        len(search_form.cleaned_data["uids"]) == 1
+        search_form.cleaned_data["initial"]
+        and len(search_form.cleaned_data["uids"]) == 1
         and search_form.cleaned_data["uids"][0] not in uid_list)
     if bad_uid:
         raise Http404


Test patch:
Repository: translate/pootle

Issue:
TP creation email should use BCC
Let's BCC by default when Pootle is communicating to lists of people.  When you create a TP all relevant people are listed in the To field.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/receivers.py b/pootle/apps/pootle_translationproject/receivers.py
--- a/pootle/apps/pootle_translationproject/receivers.py
+++ b/pootle/apps/pootle_translationproject/receivers.py
@@ -8,12 +8,12 @@
 # AUTHORS file for copyright and authorship information.
 
 from django.contrib.auth import get_user_model
-from django.core.mail import send_mail
 from django.db.models import Q
 from django.dispatch import receiver
 from django.template.loader import render_to_string
 from django.utils.translation import ugettext_lazy as _
 
+from pootle.core.mail import send_mail
 from pootle.core.url_helpers import urljoin
 
 from .models import TranslationProject
@@ -37,7 +37,7 @@ def tp_inited_async(instance, response_url, **kwargs):
     subject = _(u"Translation project (%s) created" % instance)
     recipients = get_recipients(instance.project)
     send_mail(subject, message, from_email=None,
-              recipient_list=recipients, fail_silently=True)
+              recipient_list=[], fail_silently=True, bcc=recipients)
 
 
 @receiver(tp_init_failed_async, sender=TranslationProject)
@@ -48,4 +48,4 @@ def tp_init_failed_async(instance, **kwargs):
     subject = _(u"Translation project (%s) creation failed" % instance)
     recipients = get_recipients(instance.project)
     send_mail(subject, message, from_email=None,
-              recipient_list=recipients, fail_silently=True)
+              recipient_list=[], fail_silently=True, bcc=recipients)
diff --git a/pootle/core/mail.py b/pootle/core/mail.py
--- a/pootle/core/mail.py
+++ b/pootle/core/mail.py
@@ -11,7 +11,8 @@
 
 def send_mail(subject, message, from_email, recipient_list,
               fail_silently=False, auth_user=None, auth_password=None,
-              connection=None, html_message=None, headers=None):
+              connection=None, html_message=None, headers=None,
+              cc=None, bcc=None):
     """Override django send_mail function to allow use of custom email headers.
     """
 
@@ -21,7 +22,8 @@ def send_mail(subject, message, from_email, recipient_list,
 
     mail = EmailMultiAlternatives(subject, message,
                                   from_email, recipient_list,
-                                  connection=connection, headers=headers)
+                                  connection=connection, headers=headers,
+                                  cc=cc, bcc=bcc)
 
     if html_message:
         mail.attach_alternative(html_message, 'text/html')


Test patch:
Repository: translate/pootle

Issue:
Put `Review suggestions` action button before others
Let's change order of action buttons and put `Review suggestions` before `Needs translation` to encourage going through suggestion backlog.



Discussion:
The order of columns in the project browser should also match.

So I suppose what you want in action links is:
1. **Fix critical errors**
2. **Review suggestions**
3. **Continue translation**
4. View all

And the same in the table:
1. Name
2. Progress
3. Total
4. Last updated
5. **Critical**
6. **Suggestions**
7. **Need Translation**
8. Last activity

@iafan Yes, I think it would make sense to change the order of the columns.

I think it's important to understand the why.

We'd like to order things based on priority of action.
1. Fix errors
2. Review suggestions and contributions
3. Continue with new translations

And since we're reading left to right the most important tasks should appear on the top left.

I agree with the reasoning.


Resolving patch:
diff --git a/pootle/core/browser.py b/pootle/core/browser.py
--- a/pootle/core/browser.py
+++ b/pootle/core/browser.py
@@ -50,9 +50,9 @@
         'display_name': _("Last updated"),
     },
     {
-        'id': 'need-translation',
+        'id': 'critical',
         'class': 'stats-number sorttable_numeric when-loaded',
-        'display_name': _("Need Translation"),
+        'display_name': _("Critical"),
     },
     {
         'id': 'suggestions',
@@ -61,9 +61,9 @@
         'display_name': _("Suggestions"),
     },
     {
-        'id': 'critical',
+        'id': 'need-translation',
         'class': 'stats-number sorttable_numeric when-loaded',
-        'display_name': _("Critical"),
+        'display_name': _("Need Translation"),
     },
     {
         'id': 'activity',


Test patch:
Repository: translate/pootle

Issue:
Snippet caching is not cleared between tests
Currently if you run a test that saves data in the exports cache, the data is still there in the next test



Discussion:

Resolving patch:
diff --git a/pytest_pootle/fixtures/revision.py b/pytest_pootle/fixtures/revision.py
--- a/pytest_pootle/fixtures/revision.py
+++ b/pytest_pootle/fixtures/revision.py
@@ -11,8 +11,12 @@
 
 
 @pytest.fixture(autouse=True)
-def revision():
-    """Sets up the revision counter for each test call."""
+def revision(request, clear_cache):
+    """Sets up the cached revision counter for each test call."""
     from pootle.core.models import Revision
+    from pootle_store.models import Unit
 
-    Revision.initialize()
+    if request.node.get_marker("django_db"):
+        Revision.set(Unit.max_revision())
+    else:
+        Revision.initialize()
diff --git a/pytest_pootle/fixtures/site.py b/pytest_pootle/fixtures/site.py
--- a/pytest_pootle/fixtures/site.py
+++ b/pytest_pootle/fixtures/site.py
@@ -84,3 +84,13 @@ def translations_directory(request):
     """used by PootleEnv"""
     from django.conf import settings
     settings.POOTLE_TRANSLATION_DIRECTORY = tempfile.mkdtemp()
+
+
+@pytest.fixture(autouse=True)
+def clear_cache(request):
+    """Currently tests only use one cache so this clears all"""
+
+    from django_redis import get_redis_connection
+
+    r_con = get_redis_connection('default')
+    r_con.flushdb()


Test patch:
Repository: translate/pootle

Issue:
RemovedInDjango19Warning with new commenting code
When firing up the server, Django 1.9-related warnings are printed a few times:

```
django_comments/models.py:170: RemovedInDjango19Warning: Model class django_comments.models.Comment doesn't declare an explicit app_label and either isn't in an application in INSTALLED_APPS or else was imported before its application was loaded. This will no longer be supported in Django 1.9.
  class Comment(CommentAbstractModel):

django_comments/models.py:176: RemovedInDjango19Warning: Model class django_comments.models.CommentFlag doesn't declare an explicit app_label and either isn't in an application in INSTALLED_APPS or else was imported before its application was loaded. This will no longer be supported in Django 1.9.
  class CommentFlag(models.Model):
```



Discussion:
@julen although annoying im not sure its a bug as such, it just means that these models will not work in django1.9 - but as we are not using them, i think its not a problem.

I am going to test on dj1.9 when i get chance

@phlax it does unfortunately potentially mask other issues and makes migration to Django 1.9 impossible till they're resolved :(  Same errors are being generated by Pootle FS code from what I can see.

@dwaynebailey from my understanding these are just warnings, and like i said the only problem would be if we were using these models, but were not. I could be wrong about this.

re pootle_fs for sure these need to fixed, im seeing similar problems with StaticPages, but it kinda deps on your config also

@phlax These warnings fill the logs, so any relevant error or warning will be more difficult to find. I do think that the right thing is to make these warnings go asap.


Resolving patch:
diff --git a/pootle/apps/pootle_comment/migrations/0002_use_abstract_module.py b/pootle/apps/pootle_comment/migrations/0002_use_abstract_module.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/pootle_comment/migrations/0002_use_abstract_module.py
@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+import django.db.models.deletion
+from django.conf import settings
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pootle_comment', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='comment',
+            name='user',
+            field=models.ForeignKey(related_name='comment_comments', on_delete=django.db.models.deletion.SET_NULL, verbose_name='user', blank=True, to=settings.AUTH_USER_MODEL, null=True),
+        ),
+    ]
diff --git a/pootle/apps/pootle_comment/models.py b/pootle/apps/pootle_comment/models.py
--- a/pootle/apps/pootle_comment/models.py
+++ b/pootle/apps/pootle_comment/models.py
@@ -6,7 +6,7 @@
 # or later license. See the LICENSE file for a copy of the license and the
 # AUTHORS file for copyright and authorship information.
 
-from django_comments.models import CommentAbstractModel
+from django_comments.abstracts import CommentAbstractModel
 
 
 class Comment(CommentAbstractModel):


Test patch:
Repository: translate/pootle

Issue:
Misleading label for providing administrative permissions
Administration pages where permissions can be managed include the _"can administrate a translation project"_ label. Correct me if I'm wrong, but I'm inclined to think this is misleading: if you assign such admin rights to a user, they can access the admin section(s) for the context the permissions were set. So this works site-wide and for the language, project and TP scopes, i.e. it's not limited to translation projects.



Discussion:
Yes, the permission is _"Can administrate this"_ where this can be a TP, a language or a project.

I suggest to rename this to something like "Can perform administrative tasks".


Resolving patch:
diff --git a/pootle/apps/pootle_app/migrations/0006_change_administrate_permission_name.py b/pootle/apps/pootle_app/migrations/0006_change_administrate_permission_name.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/pootle_app/migrations/0006_change_administrate_permission_name.py
@@ -0,0 +1,25 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+def change_administrate_permission_name(apps, schema_editor):
+    Permission = apps.get_model("auth", "Permission")
+
+    Permission.objects.filter(
+        codename='administrate'
+    ).update(
+        name='Can perform administrative tasks'
+    )
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pootle_app', '0005_case_sensitive_schema'),
+    ]
+
+    operations = [
+        migrations.RunPython(change_administrate_permission_name),
+    ]
diff --git a/pootle/core/initdb.py b/pootle/core/initdb.py
--- a/pootle/core/initdb.py
+++ b/pootle/core/initdb.py
@@ -146,7 +146,7 @@ def create_pootle_permissions(self):
                 'codename': "review",
             },
             {
-                'name': _("Can administrate a translation project"),
+                'name': _("Can perform administrative tasks"),
                 'codename': "administrate",
             },
         ]


Test patch:
Repository: translate/pootle

Issue:
fix update_stores not changing fuzzy state
regression from 80d35df2b2c3987470873c541c288dc95c0ec79e
ignoring change in state makes it impossible to update the fuzzy state
of a translatin unit by modifying the files on disk, and running
update_stores. With state ignored in the comparison, those changes are
silently ignored (and thenget overridden again on next sync_stores)



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/diff.py b/pootle/apps/pootle_store/diff.py
--- a/pootle/apps/pootle_store/diff.py
+++ b/pootle/apps/pootle_store/diff.py
@@ -13,14 +13,14 @@
 
 from .fields import to_python as multistring_to_python
 from .unit import UnitProxy
-from .util import OBSOLETE
+from .util import FUZZY, OBSOLETE, TRANSLATED, UNTRANSLATED
 
 
 class UnitDiffProxy(UnitProxy):
     """Wraps File/DB Unit dicts used by StoreDiff for equality comparison"""
 
     match_attrs = ["context", "developer_comment", "locations",
-                   "source", "target", "translator_comment"]
+                   "source", "state", "target", "translator_comment"]
 
     def __eq__(self, other):
         return all(getattr(self, k) == getattr(other, k)
@@ -31,7 +31,6 @@ def __ne__(self, other):
 
 
 class DBUnit(UnitDiffProxy):
-
     pass
 
 
@@ -80,13 +79,20 @@ def file_units(self):
         for unit in self.file_store.units:
             if unit.isheader():
                 continue
+            state = UNTRANSLATED
+            if unit.isobsolete():
+                state = OBSOLETE
+            elif unit.istranslated():
+                state = TRANSLATED
+            elif unit.isfuzzy():
+                state = FUZZY
             file_units[unit.getid()] = {
                 "unitid": unit.getid(),
                 "context": unit.getcontext(),
                 "locations": unit.getlocations(),
                 "source": unit.source,
                 "target": unit.target,
-                "state": unit.get_state_n(),
+                "state": state,
                 "developer_comment": unit.getnotes(origin="developer"),
                 "translator_comment": unit.getnotes(origin="translator")}
         return file_units


Test patch: