internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: translate/pootle

Issue:
pootle_fs not expiring cache_keys
When a project uses pootle FS, stats are not updated. We have to manually call `pootle flush_cache --lru --django-cache` to update it manually.


Discussion:
Copied from #5688:

As an logged in admin user:

* Create empty Pootle FS project
* View project - all blank i.e no languages
* Do `pootle fs` magic and import project
* View project, and not languages are not present.  Log you can drill down and you will find them.

We're clearly caching the old data.  I guess the same applies to any individual store that is loaded.

Resolving patch:
diff --git a/pootle/apps/pootle_revision/receivers.py b/pootle/apps/pootle_revision/receivers.py
--- a/pootle/apps/pootle_revision/receivers.py
+++ b/pootle/apps/pootle_revision/receivers.py
@@ -23,10 +23,12 @@ def handle_storedata_save(**kwargs):
 
 @receiver(post_save, sender=Directory)
 def handle_directory_save(**kwargs):
-    if kwargs.get("created"):
-        return
+    context = (
+        kwargs["instance"].parent
+        if kwargs.get("created")
+        else kwargs["instance"])
     revision_updater.get(Directory)(
-        context=kwargs["instance"]).update(keys=["stats", "checks"])
+        context=context).update(keys=["stats", "checks"])
 
 
 @receiver(pre_delete, sender=Directory)


Test patch:
Repository: translate/pootle

Issue:
Adding a new languages forces you to specify special characters
If you add a new language you are blocked with the UI insisting that you add special characters.

![screen shot 2017-01-16 at 16 01 52](https://cloud.githubusercontent.com/assets/647438/21990281/17dee6e4-dc06-11e6-910a-4b58c6c45192.png)

The part in red is Afrikaans and say "This field is required"

Not all languages require special characters.

Also the model allows `blank=True`

So somewhere we're blocking.


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/forms.py b/pootle/apps/pootle_app/forms.py
--- a/pootle/apps/pootle_app/forms.py
+++ b/pootle/apps/pootle_app/forms.py
@@ -25,7 +25,7 @@
 
 class LanguageForm(forms.ModelForm):
 
-    specialchars = forms.CharField(strip=False)
+    specialchars = forms.CharField(strip=False, required=False)
 
     class Meta(object):
         model = Language


Test patch:
diff --git a/tests/forms/language.py b/tests/forms/language.py
--- a/tests/forms/language.py
+++ b/tests/forms/language.py
@@ -12,6 +12,7 @@
 
 
 @pytest.mark.parametrize('specialchars', [
+    ' ',
     ' abcde ',
     ' ab cd',
     ' abcde',
@@ -52,3 +53,18 @@ def test_clean_specialchars_unique(specialchars, count_char):
     form = LanguageForm(form_data)
     assert form.is_valid()
     assert form.cleaned_data['specialchars'].count(count_char) == 1
+
+
+@pytest.mark.django_db
+def test_specialchars_can_be_blank():
+    """Test that a blank special character field is valid."""
+    form_data = {
+        'code': 'foo',
+        'fullname': 'Foo',
+        'checkstyle': 'foo',
+        'nplurals': '2',
+        'specialchars': '',
+    }
+    form = LanguageForm(form_data)
+    assert form.is_valid()
+    assert form.cleaned_data['specialchars'] == ''
Repository: translate/pootle

Issue:
Plural strings not marked as that
While reviewing strings for 2.8 release, those are not in plural forms:

- http://pootle.locamotion.org/ar/pootle/translate/pootle.po#unit=1826598
- http://pootle.locamotion.org/ar/pootle/translate/pootle.po#unit=1826597
- http://pootle.locamotion.org/ar/pootle/translate/pootle.po#unit=1826594
- http://pootle.locamotion.org/ar/pootle/translate/pootle.po#unit=1826584


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_language/views.py b/pootle/apps/pootle_language/views.py
--- a/pootle/apps/pootle_language/views.py
+++ b/pootle/apps/pootle_language/views.py
@@ -23,7 +23,7 @@
 from pootle.core.views.formtable import Formtable
 from pootle.core.views.mixins import PootleJSONMixin
 from pootle.i18n import formatter
-from pootle.i18n.gettext import tr_lang, ugettext_lazy as _
+from pootle.i18n.gettext import tr_lang, ugettext_lazy as _, ungettext_lazy
 from pootle_misc.util import cmp_by_last_activity
 from pootle_store.constants import STATES_MAP
 
@@ -232,17 +232,25 @@ def add_success_message(self, form):
             form.cleaned_data["actions"] == "accept"
             and form.cleaned_data["comment"])
         if reject_and_notify:
-            message = _(
+            message = ungettext_lazy(
+                "Rejected %s suggestion with comment. User will be notified",
                 "Rejected %s suggestions with comment. Users will be notified",
                 count)
         elif accept_and_notify:
-            message = _(
+            message = ungettext_lazy(
+                "Accepted %s suggestion with comment. User will be notified",
                 "Accepted %s suggestions with comment. Users will be notified",
                 count)
         elif form.cleaned_data["actions"] == "reject":
-            message = _("Rejected %s suggestions", count)
+            message = ungettext_lazy(
+                "Rejected %s suggestion",
+                "Rejected %s suggestions",
+                count)
         else:
-            message = _("Accepted %s suggestions", count)
+            message = ungettext_lazy(
+                "Accepted %s suggestion",
+                "Accepted %s suggestions",
+                count)
         messages.success(self.request, message)
 
     def get_context_data(self, **kwargs):


Test patch:
Repository: translate/pootle

Issue:
Templates language is taken into account in the main view
I'm testing Pootle 2.8.0RC3 and I've found an issue related to #4568.

When I filter the translations for a single project, the progress bar now shows 100% (the templates aren't taken into account now, great):
![image](https://cloud.githubusercontent.com/assets/104028/22153800/1b63c99e-df29-11e6-903a-f944c4375213.png)

However, when I go back to the global view, that project shows a progress bar including the templates result:
![image](https://cloud.githubusercontent.com/assets/104028/22153805/21d6f4c2-df29-11e6-9134-2e39ce3583c7.png)

Thank you!


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_data/project_data.py b/pootle/apps/pootle_data/project_data.py
--- a/pootle/apps/pootle_data/project_data.py
+++ b/pootle/apps/pootle_data/project_data.py
@@ -62,3 +62,7 @@ def get_root_child_path(self, child):
     @property
     def context_name(self):
         return "ALL"
+
+    def filter_data(self, qs):
+        qs = super(ProjectSetDataTool, self).filter_data(qs)
+        return qs.exclude(tp__language__code="templates")


Test patch:
diff --git a/tests/pootle_data/data_tool.py b/tests/pootle_data/data_tool.py
--- a/tests/pootle_data/data_tool.py
+++ b/tests/pootle_data/data_tool.py
@@ -356,6 +356,7 @@ def test_data_project_store_stats(project_store_resources0):
 def test_data_project_set_stats(project_set):
     units = Unit.objects.live().exclude(
         store__translation_project__project__disabled=True)
+    units = units.exclude(store__is_template=True)
     stats = project_set.data_tool.get_stats(include_children=False)
     _test_object_stats(
         stats,
Repository: translate/pootle

Issue:
POOTLE_FS_WORKING_PATH setting hint is unhelpful without the directory
```console
?: (pootle.C021) Path pointed to by POOTLE_FS_WORKING_PATH is not writable by Pootle.
        HINT: Add the write permission to the `POOTLE_FS_WORKING_PATH` directory using 'chmod +w'
```

The hint is unhelpful without the actual path being pointed to by `POOTLE_FS_WORKING_PATH`


Discussion:
the other thing im thinking is that the advices is to chmod the dir - whereas it might make more sense to change the setting
and advising with chmod assumes unix-based system (iirc)

Resolving patch:
diff --git a/pootle/checks.py b/pootle/checks.py
--- a/pootle/checks.py
+++ b/pootle/checks.py
@@ -503,16 +503,18 @@ def check_pootle_fs_working_dir(app_configs=None, **kwargs):
         id="pootle.C019",
     )
     missing_directory_error = checks.Critical(
-        _("Path pointed to by POOTLE_FS_WORKING_PATH doesn't exist."),
-        hint=_("Create the directory pointed by `POOTLE_FS_WORKING_PATH` "
-               "setting."),
+        _("Path ('%s') pointed to by POOTLE_FS_WORKING_PATH doesn't exist."
+          % settings.POOTLE_FS_WORKING_PATH),
+        hint=_("Create the directory pointed by `POOTLE_FS_WORKING_PATH`, "
+               "or change the setting."),
         id="pootle.C020",
     )
     not_writable_directory_error = checks.Critical(
-        _("Path pointed to by POOTLE_FS_WORKING_PATH is not writable by "
-          "Pootle."),
+        _("Path ('%s') pointed to by POOTLE_FS_WORKING_PATH is not writable by "
+          "Pootle."
+          % settings.POOTLE_FS_WORKING_PATH),
         hint=_("Add the write permission to the `POOTLE_FS_WORKING_PATH` "
-               "directory using 'chmod +w'"),
+               "or change the setting"),
         id="pootle.C021",
     )
     errors = []


Test patch:
Repository: translate/pootle

Issue:
Add language on pootle_fs/xliff support
Ubuntu 16.04
Pootle 2.8.0b3

Have succesfully created a pootle_fs project.
Based on xliff, I put a file with my GNU style "messages.<language_code>.<ext>"
I tried french and templates as language, both are taken, uploaded and I can translate.
But when I add any language, my RQworker get me this error in both language :

```
[2016-09-13T10:13:35]   default: pootle_project.forms.update_translation_project(<TranslationProject: /sq/messages/>, True, 'http://localhost:8000/') (cfeaa7eb-99ae-4e4f-bbab-3cfa0e96d436)
2016-09-13 10:13:55,524 INFO Scanning for new files in /sq/messages/
[2016-09-13T10:14:05]   AttributeError: 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):
  File "/home/valentin/dev/pootle/env/local/lib/python2.7/site-packages/rq/worker.py", line 568, in perform_job
    rv = job.perform()
  File "/home/valentin/dev/pootle/env/local/lib/python2.7/site-packages/rq/job.py", line 495, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/home/valentin/dev/pootle/env/local/lib/python2.7/site-packages/pootle/apps/pootle_project/forms.py", line 45, in update_translation_project
    raise e
AttributeError: 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):
  File "/home/valentin/dev/pootle/env/local/lib/python2.7/site-packages/rq/worker.py", line 568, in perform_job
    rv = job.perform()
  File "/home/valentin/dev/pootle/env/local/lib/python2.7/site-packages/rq/job.py", line 495, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/home/valentin/dev/pootle/env/local/lib/python2.7/site-packages/pootle/apps/pootle_project/forms.py", line 45, in update_translation_project
    raise e
AttributeError: 'NoneType' object has no attribute 'startswith'
```



Discussion:
@phlax Do you have any input on this one?

Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/models.py b/pootle/apps/pootle_translationproject/models.py
--- a/pootle/apps/pootle_translationproject/models.py
+++ b/pootle/apps/pootle_translationproject/models.py
@@ -438,7 +438,10 @@ def scan_projects(**kwargs):
     if not created or raw:
         return
 
-    for project in Project.objects.enabled().iterator():
+    old_style_projects = Project.objects.enabled().exclude(
+        treestyle="pootle_fs")
+
+    for project in old_style_projects.iterator():
         tp = create_translation_project(instance, project)
         if tp is not None:
             tp.update_from_disk()


Test patch:
Repository: translate/pootle

Issue:
Can't delete any last TP on languages page in admin UI
If project is not pootle_fs and doesn't have templates then it's impossible to delete the last TP (checkbox is missing).
![image](https://cloud.githubusercontent.com/assets/682845/22365776/690a48e2-e48a-11e6-8069-7aec862a4e99.png)



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/views/admin/util.py b/pootle/apps/pootle_app/views/admin/util.py
--- a/pootle/apps/pootle_app/views/admin/util.py
+++ b/pootle/apps/pootle_app/views/admin/util.py
@@ -112,7 +112,8 @@ def add_widgets(result, fields, form, link):
         result.append('<tbody>\n')
 
         # Do not display the delete checkbox for the 'add a new entry' form.
-        formset.forms[-1].fields['DELETE'].widget = forms.HiddenInput()
+        if formset.extra_forms:
+            formset.forms[-1].fields['DELETE'].widget = forms.HiddenInput()
 
         for form in formset.forms:
             add_errors(result, fields, form)


Test patch:
Repository: translate/pootle

Issue:
Dates not getting localised in browse tables
seems like the dates are not getting localised as they should


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/panels.py b/pootle/apps/pootle_app/panels.py
--- a/pootle/apps/pootle_app/panels.py
+++ b/pootle/apps/pootle_app/panels.py
@@ -58,11 +58,15 @@ def child_update_times(self):
             if not child.get("stats"):
                 continue
             last_created_unit = (
-                timesince(child["stats"]["last_created_unit"]["creation_time"])
+                timesince(
+                    child["stats"]["last_created_unit"]["creation_time"],
+                    locale=self.view.request_lang)
                 if child["stats"].get("last_created_unit")
                 else None)
             last_submission = (
-                timesince(child["stats"]["last_submission"]["mtime"])
+                timesince(
+                    child["stats"]["last_submission"]["mtime"],
+                    locale=self.view.request_lang)
                 if child["stats"].get("last_submission")
                 else None)
             _times[child["code"]] = (last_submission, last_created_unit)
diff --git a/pootle/i18n/dates.py b/pootle/i18n/dates.py
--- a/pootle/i18n/dates.py
+++ b/pootle/i18n/dates.py
@@ -49,4 +49,6 @@ def format_timesince(self, timestamp, locale=None):
 
 
 def timesince(timestamp, locale=None):
+    if locale:
+        locale = translation.to_locale(locale)
     return localdate.format_timesince(timestamp, locale=locale)


Test patch:
Repository: translate/pootle

Issue:
Setting FORCE_SCRIPT_NAME breaks views
Seems like a number of views are not parsing path correctly when FORCE_SCRIPT_NAME is set.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -291,6 +291,15 @@ class TPBrowseView(TPDirectoryMixin, TPBrowseBaseView):
     view_name = "tp"
     panel_names = ('vfolders', 'children')
 
+    @property
+    def path(self):
+        kwargs = self.kwargs
+        kwargs["dir_path"] = kwargs.get("dir_path", "")
+        kwargs["filename"] = kwargs.get("filename", "")
+        return (
+            "/%(language_code)s/%(project_code)s/%(dir_path)s%(filename)s"
+            % kwargs)
+
     @cached_property
     def object_children(self):
         dirs_with_vfolders = []


Test patch:
Repository: translate/pootle

Issue:
TMX export incorrectly names files
the tmx files are getting a .zip extension inside the zip archive


Discussion:

Resolving patch:
diff --git a/pootle/apps/import_export/utils.py b/pootle/apps/import_export/utils.py
--- a/pootle/apps/import_export/utils.py
+++ b/pootle/apps/import_export/utils.py
@@ -167,7 +167,7 @@ def export(self, rotate=False):
         tmxfile.serialize(bs)
         with open(self.abs_filepath, "wb") as f:
             with ZipFile(f, "w") as zf:
-                zf.writestr(self.filename, bs.getvalue())
+                zf.writestr(self.filename.rstrip('.zip'), bs.getvalue())
 
         last_exported_filepath = self.last_exported_file_path
         self.update_exported_revision()


Test patch:
Repository: translate/pootle

Issue:
IntegrityError: (1062, "Duplicate entry 'xxx-stats' for key > 'pootle_revision_content_type_id_xxx_uniq'")
This error has been spotted in the wild.

From code review its hard to see how its happening, and i ~~havent~~ managed to reproduce - *by clicking very fast on editor buttons*

~~It may be related to update_stores~~


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_revision/utils.py b/pootle/apps/pootle_revision/utils.py
--- a/pootle/apps/pootle_revision/utils.py
+++ b/pootle/apps/pootle_revision/utils.py
@@ -169,7 +169,10 @@ def create_revisions(self, parents, keys=None):
     def update(self, keys=None):
         parents = list(self.parents.values_list("id", flat=True))
         revisions = self.get_revisions(parents, keys=keys)
-        revisions.delete()
+        # manually get the list of ids and delete those to prevent
+        # django race condition
+        revision_ids = list(revisions.values_list("id", flat=True))
+        revisions.filter(id__in=revision_ids).delete()
         Revision.objects.bulk_create(
             self.create_revisions(parents, keys=keys))
 


Test patch:
Repository: translate/pootle

Issue:
Form validation on login form exposes variables
if you try to signup with an existing username you get this

```
%(model_name)s with this %(field_label)s already exists.
```


Discussion:

Resolving patch:
diff --git a/pootle/apps/accounts/migrations/0010_set_username_unique_error_message.py b/pootle/apps/accounts/migrations/0010_set_username_unique_error_message.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/accounts/migrations/0010_set_username_unique_error_message.py
@@ -0,0 +1,22 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.10.6 on 2017-03-03 10:47
+from __future__ import unicode_literals
+
+import django.core.validators
+from django.db import migrations, models
+import re
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('accounts', '0009_remove_currency_and_hourly_rate_fields'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='user',
+            name='username',
+            field=models.CharField(error_messages={b'unique': 'A user with that username already exists.'}, help_text='Required. 30 characters or fewer. Letters, numbers and @/./+/-/_ characters', max_length=30, unique=True, validators=[django.core.validators.RegexValidator(re.compile(b'^[\\w.@+-]+$', 32), 'Enter a valid username.', b'invalid')], verbose_name='Username'),
+        ),
+    ]
diff --git a/pootle/apps/accounts/models.py b/pootle/apps/accounts/models.py
--- a/pootle/apps/accounts/models.py
+++ b/pootle/apps/accounts/models.py
@@ -10,7 +10,7 @@
 from hashlib import md5
 
 from django.conf import settings
-from django.contrib.auth.models import AbstractBaseUser
+from django.contrib.auth.base_user import AbstractBaseUser
 from django.core.exceptions import ValidationError
 from django.core.mail import send_mail
 from django.core.validators import RegexValidator
@@ -60,6 +60,9 @@ class User(AbstractBaseUser):
                            _('Enter a valid username.'),
                            'invalid')
         ],
+        error_messages={
+            'unique': _('A user with that username already exists.'),
+        },
     )
     email = models.EmailField(_('Email Address'), max_length=255)
     full_name = models.CharField(_('Full Name'), max_length=255, blank=True)


Test patch:
Repository: translate/pootle

Issue:
Delete a TP from an old style project and the Project page stays cached
1. Create a new TP
2. TP is available
3. Delete TP
4. Project page still shows project listed - though it should be gone
5. Going to supposedly deleted TP and we get 404

We're not expiring cache when a TP is deleted.


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_revision/receivers.py b/pootle/apps/pootle_revision/receivers.py
--- a/pootle/apps/pootle_revision/receivers.py
+++ b/pootle/apps/pootle_revision/receivers.py
@@ -13,6 +13,7 @@
 from pootle_app.models import Directory
 from pootle_data.models import StoreData
 from pootle_store.models import Store
+from pootle_translationproject.models import TranslationProject
 
 
 @receiver(post_save, sender=StoreData)
@@ -35,3 +36,9 @@ def handle_directory_save(**kwargs):
 def handle_directory_delete(**kwargs):
     revision_updater.get(Directory)(
         context=kwargs["instance"].parent).update(keys=["stats", "checks"])
+
+
+@receiver(pre_delete, sender=TranslationProject)
+def handle_tp_delete(**kwargs):
+    revision_updater.get(Directory)(
+        context=kwargs["instance"].directory).update(keys=["stats", "checks"])


Test patch:
diff --git a/tests/models/translationproject.py b/tests/models/translationproject.py
--- a/tests/models/translationproject.py
+++ b/tests/models/translationproject.py
@@ -17,6 +17,7 @@
 from pytest_pootle.factories import (
     LanguageDBFactory, ProjectDBFactory, TranslationProjectFactory)
 
+from pootle.core.delegate import revision
 from pootle_language.models import Language
 from pootle_project.models import Project
 from pootle_store.models import Store
@@ -175,3 +176,15 @@ def test_tp_create_with_none_treestyle(po_directory, english, templates, setting
         os.path.join(
             settings.POOTLE_TRANSLATION_DIRECTORY,
             project.code))
+
+
+@pytest.mark.django_db
+def test_tp_cache_on_delete(tp0):
+    proj_revision = revision.get(
+        tp0.project.directory.__class__)(
+            tp0.project.directory)
+    orig_revision = proj_revision.get("stats")
+    tp0.delete()
+    assert (
+        proj_revision.get("stats")
+        != orig_revision)
Repository: translate/pootle

Issue:
String needs to use plural form
Reported by @milupo in https://github.com/translate/pootle/issues/6061#issuecomment-284076850

There is one string more:

templates/includes/formtable.html:27
Showing %(count)s results per page

I know, percents are not used here.


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/forms.py b/pootle/apps/pootle_store/forms.py
--- a/pootle/apps/pootle_store/forms.py
+++ b/pootle/apps/pootle_store/forms.py
@@ -540,7 +540,7 @@ def clean_action(self):
             self.add_error(
                 "action",
                 forms.ValidationError(
-                    _("Suggestion '%s' cannot be accepted/rejected twice!",
+                    _("Suggestion '%s' has already been accepted or rejected.",
                       self.target_object)))
         return self.data["action"]
 


Test patch: