internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: translate/pootle

Issue:
`sync_stores` doesn't handle disabled projects
We addressed the similar issue for `update_stores` #4198.
`sync_stores` should work for disabled projects as well https://github.com/translate/pootle/issues/4198#issuecomment-161717337.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/management/commands/sync_stores.py b/pootle/apps/pootle_app/management/commands/sync_stores.py
--- a/pootle/apps/pootle_app/management/commands/sync_stores.py
+++ b/pootle/apps/pootle_app/management/commands/sync_stores.py
@@ -40,13 +40,15 @@ class Command(PootleCommand):
         ),
     )
     help = "Save new translations to disk manually."
+    process_disabled_projects = True
 
     def handle_all_stores(self, translation_project, **options):
-        translation_project.sync(
-            conservative=not options['overwrite'],
-            skip_missing=options['skip_missing'],
-            only_newer=not options['force']
-        )
+        if translation_project.directory_exists_on_disk():
+            translation_project.sync(
+                conservative=not options['overwrite'],
+                skip_missing=options['skip_missing'],
+                only_newer=not options['force']
+            )
 
     def handle_store(self, store, **options):
         store.sync(


Test patch:
Repository: translate/pootle

Issue:
Editor templates broken in non-debug mode
Since switching to Django 1.8 in master, the [scripts section of the editor templates](https://github.com/translate/pootle/blob/master/pootle/templates/editor/_scripts.html) doesn't render when `DEBUG = False`.

I might be doing something wrong, but I tried removing any template customizations we have, also clearing out caches, trying different browsers, always with the same result. I can reproduce this locally and in our staging server.



Discussion:
@julen we have the same problem on MLO - I tested against default install and it seemed to work ok

im not sure of exactly the problem, but for some reason the browser/index.html template is not being included by overextends

If this isn't an easy fix, can we undo switching to Django 1.8 until this is resolved? I'd prefer the related commits to be moved to a side feature branch so that master is stable.

@iafan leave this with me - i reckon we can resolve it one way or another

Thanks, @phlax! This is quite urgent for us as prevents testing and a release, so hope to hear some news from you in this soon.


Resolving patch:
diff --git a/pootle/apps/pootle_store/templatetags/store_tags.py b/pootle/apps/pootle_store/templatetags/store_tags.py
--- a/pootle/apps/pootle_store/templatetags/store_tags.py
+++ b/pootle/apps/pootle_store/templatetags/store_tags.py
@@ -234,7 +234,8 @@ def do_include_raw(parser, token):
             template_name[-1] == template_name[0]):
         template_name = template_name[1:-1]
 
-    source, path = get_template(
-        template_name).origin.loader(template_name)
-
-    return template.base.TextNode(source)
+    return template.base.TextNode(
+        u"\n".join(
+            [x.s
+             for x
+             in get_template(template_name).template.nodelist]))


Test patch:
Repository: translate/pootle

Issue:
Stats: dirty banner can appear after refresh_stats or any translation actions with rq==0.5.6
`refresh_stats` finishes without failures but produces some incomplete stats and the dirty banner visible.
I reckon that it happens because jobs with dependencies are added to the wrong queue `rq:queue:None`. This happened after `rq==0.5.6` version was installed.



Discussion:

Resolving patch:
diff --git a/pootle/core/mixins/treeitem.py b/pootle/core/mixins/treeitem.py
--- a/pootle/core/mixins/treeitem.py
+++ b/pootle/core/mixins/treeitem.py
@@ -650,7 +650,7 @@ def create_job(self, status=None, depends_on=None):
         args = (self.instance,)
         return Job.create(self.func, args=args, id=self.id,
                           connection=self.connection, depends_on=depends_on,
-                          status=status)
+                          status=status, origin=self.origin)
 
     def save_enqueued(self, pipe):
         """


Test patch:
Repository: translate/pootle

Issue:
Accessing the timeline provokes a server error
Accessing the timeline currently provokes a server error:

``` python
FieldError: Invalid field name(s) given in select_related: 'user'. Choices are: evernote_account
  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "pootle_store/decorators.py", line 66, in decorated_f
    return f(request, unit, *args, **kwargs)
  File "pootle_store/views.py", line 604, in timeline
    key=lambda x: "%d\001%s" % (x.submitter.id, x.creation_time)):
  File "django/db/models/query.py", line 162, in __iter__
    self._fetch_all()
  File "django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "django/db/models/sql/compiler.py", line 829, in execute_sql
    sql, params = self.as_sql()
  File "django/db/models/sql/compiler.py", line 378, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File "django/db/models/sql/compiler.py", line 48, in pre_sql_setup
    self.setup_query()
  File "django/db/models/sql/compiler.py", line 39, in setup_query
    self.select, self.klass_info, self.annotation_col_map = self.get_select()
  File "django/db/models/sql/compiler.py", line 206, in get_select
    related_klass_infos = self.get_related_selections(select)
  File "django/db/models/sql/compiler.py", line 708, in get_related_selections
    select, f.rel.to._meta, alias, cur_depth + 1, next, restricted)
  File "django/db/models/sql/compiler.py", line 754, in get_related_selections
    ', '.join(_get_field_choices()) or '(none)',
```

I believe this is [`select_related()` checking for actual fields](https://docs.djangoproject.com/en/1.9/releases/1.8/#select-related-now-checks-given-fields) since Django 1.8+, however the [prohibition is put in place in Django 1.10](https://docs.djangoproject.com/en/dev/releases/1.10/#select-related-prohibits-non-relational-fields-for-nested-relations).



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -590,7 +590,7 @@ def timeline(request, unit):
         field=SubmissionFields.COMMENT,
         creation_time=unit.commented_on
     ).order_by("id")
-    timeline = timeline.select_related("submitter__user",
+    timeline = timeline.select_related("submitter",
                                        "translation_project__language")
 
     User = get_user_model()


Test patch:
Repository: translate/pootle

Issue:
[Regression] Language codes below TPs are case-sensitive
Case insensitivity in language codes has regressed when accessing translation projects as reported by our L10N managers.

While from the user perspective both `/pt_BR/` and `/pt_br/` work and point to the same location, accessing a translation project produces inconsistent results: `/pt_BR/<project_code>/` is accessible, whereas `/pt_br/<project_code>/` is not.
_(For the sake of completeness, the actual case for the language code in the server is `pt_BR`)_

This can be reproduced in EN's production server:
- https://translate.evernote.com/pt_BR/
- https://translate.evernote.com/pt_br/
- https://translate.evernote.com/pt_BR/corenote_evernote/
- https://translate.evernote.com/pt_br/corenote_evernote/ **(404)**

While the previous behavior can be reproduced in MLO:
- https://mozilla.locamotion.org/pt_BR/
- https://mozilla.locamotion.org/pt_br/
- https://mozilla.locamotion.org/pt_BR/terminology/
- https://mozilla.locamotion.org/pt_br/terminology/

Accessing resources under the lowercase language code also suffers from this issue.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_language/views.py b/pootle/apps/pootle_language/views.py
--- a/pootle/apps/pootle_language/views.py
+++ b/pootle/apps/pootle_language/views.py
@@ -42,7 +42,7 @@ def url_kwargs(self):
 
     def get_object(self):
         return get_object_or_404(
-            Language, code=self.kwargs["language_code"])
+            Language, code__iexact=self.kwargs["language_code"])
 
 
 class LanguageBrowseView(LanguageMixin, PootleBrowseView):
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -72,7 +72,7 @@ def method_wrapper(self, request, *args, **kwargs):
                 try:
                     TranslationProject.objects.get(
                         project__code=kwargs["project_code"],
-                        language__code=kwargs["language_code"])
+                        language__code__iexact=kwargs["language_code"])
                     # the TP exists so redirect to it
                     return redirect(
                         reverse(


Test patch:
Repository: translate/pootle

Issue:
Do not display unnecessary permissions
When assigning permissions for users several unnecessary permissions are listed:
- `Can add directory`
- `Can change directory`
- `Can delete directory`

This happens for TP, language and project, and probably in all the permission views.

Expected behavior is not to list those in order to do not overwhelm with long permission choices.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/migrations/0003_drop_existing_directory_default_permissions.py b/pootle/apps/pootle_app/migrations/0003_drop_existing_directory_default_permissions.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/pootle_app/migrations/0003_drop_existing_directory_default_permissions.py
@@ -0,0 +1,23 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+def drop_existing_directory_default_permissions(apps, schema_editor):
+    Permission = apps.get_model("auth", "Permission")
+
+    Permission.objects.filter(
+        codename__in=['add_directory', 'change_directory', 'delete_directory']
+    ).delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pootle_app', '0002_mark_empty_dirs_as_obsolete'),
+    ]
+
+    operations = [
+        migrations.RunPython(drop_existing_directory_default_permissions),
+    ]
diff --git a/pootle/apps/pootle_app/migrations/0004_set_directory_has_no_default_permissions.py b/pootle/apps/pootle_app/migrations/0004_set_directory_has_no_default_permissions.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/pootle_app/migrations/0004_set_directory_has_no_default_permissions.py
@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pootle_app', '0003_drop_existing_directory_default_permissions'),
+    ]
+
+    operations = [
+        migrations.AlterModelOptions(
+            name='directory',
+            options={'ordering': ['name'], 'default_permissions': ()},
+        ),
+    ]
diff --git a/pootle/apps/pootle_app/models/directory.py b/pootle/apps/pootle_app/models/directory.py
--- a/pootle/apps/pootle_app/models/directory.py
+++ b/pootle/apps/pootle_app/models/directory.py
@@ -49,6 +49,7 @@ class Directory(models.Model, CachedTreeItem):
 
     class Meta(object):
         ordering = ['name']
+        default_permissions = ()
         app_label = "pootle_app"
 
     # # # # # # # # # # # # # #  Properties # # # # # # # # # # # # # # # # # #


Test patch:
Repository: translate/pootle

Issue:
Coalesce in get_units is creating bad sql
Im seeing errors in get_units, regarding bad sql

this is the dump of the sql in question, seems like theres a bracket missing - not sure how that has happened, but im figuring something to do with COALESCE

``` sql
SELECT DISTINCT `pootle_store_unit`.`id`, `pootle_store_store`.`pootle_path`,                         ) FROM `pootle_store_unit` INNER JOIN `pootle_store_store` ON ( `pootle_store_unit`.`store_id` = `pootle_store_store`.`id` ) INNER JOIN `pootle_app_translationproject` ON ( `pootle_store_store`.`translation_project_id` = `pootle_app_translationproject`.`id` ) INNER JOIN `pootle_app_project` ON ( `pootle_app_translationproject`.`project_id` = `pootle_app_project`.`id` ) INNER JOIN `pootle_app_language` ON ( `pootle_app_translationproject`.`language_id` = `pootle_app_language`.`id` ) INNER JOIN `pootle_store_qualitycheck` ON ( `pootle_store_unit`.`id` = `pootle_store_qualitycheck`.`unit_id` ) WHERE (`pootle_store_unit`.`state` > -100 AND `pootle_app_project`.`disabled` = False AND `pootle_app_language`.`code` = ka AND `pootle_app_project`.`code` = firefox AND `pootle_store_store`.`pootle_path` LIKE BINARY /ka/firefox/toolkit/% AND `pootle_store_qualitycheck`.`category` = 100 AND `pootle_store_qualitycheck`.`false_positive` = False) ORDER BY (
                            SELECT
                              COALESCE(MAX(virtualfolder_virtualfolder.priority), 1)
                            FROM virtualfolder_virtualfolder
                            INNER JOIN virtualfolder_virtualfolder_units
                            ON virtualfolder_virtualfolder.id =
                              virtualfolder_virtualfolder_units.virtualfolder_id
                            WHERE virtualfolder_virtualfolder_units.unit_id =
                              pootle_store_unit.id
                        ) DESC LIMIT 21

```

This will be fixed in the move to a search backend, but im thinking that as we no longer need to do this (vfolder.priority is denormalized) we should fix first



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -287,8 +287,7 @@ def get_step_query(request, units_queryset):
             sort_by = ALLOWED_SORTS[sort_on].get(sort_by_param, None)
             if sort_by is not None:
                 if sort_on in SIMPLY_SORTED:
-                    if sort_by == 'priority':
-                        match_queryset = match_queryset.order_by(sort_by)
+                    match_queryset = match_queryset.order_by(sort_by)
                 else:
                     # Omit leading `-` sign
                     if sort_by[0] == '-':


Test patch:
Repository: translate/pootle

Issue:
Coalesce in get_units is creating bad sql
Im seeing errors in get_units, regarding bad sql

this is the dump of the sql in question, seems like theres a bracket missing - not sure how that has happened, but im figuring something to do with COALESCE

``` sql
SELECT DISTINCT `pootle_store_unit`.`id`, `pootle_store_store`.`pootle_path`,                         ) FROM `pootle_store_unit` INNER JOIN `pootle_store_store` ON ( `pootle_store_unit`.`store_id` = `pootle_store_store`.`id` ) INNER JOIN `pootle_app_translationproject` ON ( `pootle_store_store`.`translation_project_id` = `pootle_app_translationproject`.`id` ) INNER JOIN `pootle_app_project` ON ( `pootle_app_translationproject`.`project_id` = `pootle_app_project`.`id` ) INNER JOIN `pootle_app_language` ON ( `pootle_app_translationproject`.`language_id` = `pootle_app_language`.`id` ) INNER JOIN `pootle_store_qualitycheck` ON ( `pootle_store_unit`.`id` = `pootle_store_qualitycheck`.`unit_id` ) WHERE (`pootle_store_unit`.`state` > -100 AND `pootle_app_project`.`disabled` = False AND `pootle_app_language`.`code` = ka AND `pootle_app_project`.`code` = firefox AND `pootle_store_store`.`pootle_path` LIKE BINARY /ka/firefox/toolkit/% AND `pootle_store_qualitycheck`.`category` = 100 AND `pootle_store_qualitycheck`.`false_positive` = False) ORDER BY (
                            SELECT
                              COALESCE(MAX(virtualfolder_virtualfolder.priority), 1)
                            FROM virtualfolder_virtualfolder
                            INNER JOIN virtualfolder_virtualfolder_units
                            ON virtualfolder_virtualfolder.id =
                              virtualfolder_virtualfolder_units.virtualfolder_id
                            WHERE virtualfolder_virtualfolder_units.unit_id =
                              pootle_store_unit.id
                        ) DESC LIMIT 21

```

This will be fixed in the move to a search backend, but im thinking that as we no longer need to do this (vfolder.priority is denormalized) we should fix first



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -55,7 +55,7 @@
 #: will be used against the DB.
 ALLOWED_SORTS = {
     'units': {
-        'priority': 'priority',
+        'priority': '-priority',
         'oldest': 'submitted_on',
         'newest': '-submitted_on',
     },


Test patch:
Repository: translate/pootle

Issue:
Do not show disabled projects for non-admins
Not sure if this is a regression, but now, if one goes to a disabled project page, e.g. https://translate.evernote.com/projects/kb_evernote/ as a regular/anonymous user, they will see that project page header (in semi-broken state) with the message intended to be seen by admins only:

`This project is disabled and visible to administrators only.`

Expected is that such people should get a 404 page instead.

The same applies to a TP page, e.g. https://translate.evernote.com/ru/kb_evernote/
and to folders: https://translate.evernote.com/ru/kb_evernote/articles/

As it turns out, people **can actually see the contents of such disabled projects**, which makes this issue a priority one.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_project/views.py b/pootle/apps/pootle_project/views.py
--- a/pootle/apps/pootle_project/views.py
+++ b/pootle/apps/pootle_project/views.py
@@ -49,9 +49,12 @@ def permission_context(self):
 
     @cached_property
     def project(self):
-        return get_object_or_404(
+        project = get_object_or_404(
             Project.objects.select_related("directory"),
             code=self.kwargs["project_code"])
+        if project.disabled and not self.request.profile.is_superuser:
+            raise Http404
+        return project
 
     @property
     def url_kwargs(self):
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -143,6 +143,8 @@ def tp(self):
 
     @cached_property
     def project(self):
+        if self.tp.project.disabled and not self.request.profile.is_superuser:
+            raise Http404
         return self.tp.project
 
     @cached_property


Test patch:
Repository: translate/pootle

Issue:
Download and upload buttons don't appear for a single file
When you browser into a [single file](https://mozilla.locamotion.org/en_ZA/firefox/browser/branding/official/brand.dtd.po) you don't have access to any of the upload and download links.

Thus if you want a single files in a large directory you have to [download the whole directory](https://mozilla.locamotion.org/en_ZA/firefox/browser/branding/official/).



Discussion:
This might be a regression from when views have been splitted https://github.com/translate/pootle/commit/02a24f71e40ec63701087bb993414c14280eb402.

I think this can be fixed by moving `get_upload_widget` to `TPBrowseBaseView`.


Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -228,10 +228,24 @@ class TPBrowseBaseView(PootleBrowseView):
 
     def get_context_data(self, *args, **kwargs):
         ctx = super(TPBrowseBaseView, self).get_context_data(*args, **kwargs)
+        ctx.update(self.get_upload_widget(self.project))
         ctx.update(
             {'parent': get_parent(self.object)})
         return ctx
 
+    def get_upload_widget(self, project):
+        ctx = {}
+        has_upload = (
+            "import_export" in settings.INSTALLED_APPS
+            and self.request.user.is_authenticated()
+            and check_permission('translate', self.request))
+        if has_upload:
+            ctx.update(handle_upload_form(self.request, project))
+            ctx.update(
+                {'display_download': True,
+                 'has_sidebar': True})
+        return ctx
+
 
 class TPBrowseStoreView(TPStoreMixin, TPBrowseBaseView):
     pass
@@ -265,19 +279,6 @@ def items(self):
             if isinstance(child, Store)]
         return directories + stores
 
-    def get_upload_widget(self, project):
-        ctx = {}
-        has_upload = (
-            "import_export" in settings.INSTALLED_APPS
-            and self.request.user.is_authenticated()
-            and check_permission('translate', self.request))
-        if has_upload:
-            ctx.update(handle_upload_form(self.request, project))
-            ctx.update(
-                {'display_download': True,
-                 'has_sidebar': True})
-        return ctx
-
     @cached_property
     def has_vfolders(self):
         return self.object.has_vfolders
@@ -334,7 +335,6 @@ def stats(self):
 
     def get_context_data(self, *args, **kwargs):
         ctx = super(TPBrowseView, self).get_context_data(*args, **kwargs)
-        ctx.update(self.get_upload_widget(self.project))
         ctx.update(self.vfolder_data)
         return ctx
 


Test patch: