internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: translate/pootle

Issue:
Enable project resource pages (across all languages) for disabled projects for admins
The change in #3997 allowed admins to browse through disabled projects but project resource pages are still unavailable.



Discussion:

Resolving patch:
diff --git a/pootle/core/decorators.py b/pootle/core/decorators.py
--- a/pootle/core/decorators.py
+++ b/pootle/core/decorators.py
@@ -208,16 +208,17 @@ def set_project_resource(request, path_obj, dir_path, filename):
     resource_path = dir_path
     pootle_path = ctx_path + dir_path
 
-    # List of disabled TP paths
-    disabled_tps = TranslationProject.objects.disabled().filter(
+    # List of TP paths available for user
+    user_tps = TranslationProject.objects.for_user(request.user)
+    user_tps = user_tps.filter(
         project__code=path_obj.code,
     ).values_list('pootle_path', flat=True)
-    disabled_tps = list(disabled_tps)
-    disabled_tps.append('/templates/')
-    disabled_tps_regex = '^%s' % u'|'.join(disabled_tps)
-    sql_not_regex = 'NOT REGEXP'
+    user_tps = list(path for path in user_tps
+                    if not path.startswith('/templates/'))
+    user_tps_regex = '^%s' % u'|'.join(user_tps)
+    sql_regex = 'REGEXP'
     if connection.vendor == 'postgresql':
-        sql_not_regex = '!~'
+        sql_regex = '~'
 
     if filename:
         query_pootle_path = query_pootle_path + filename
@@ -227,15 +228,15 @@ def set_project_resource(request, path_obj, dir_path, filename):
         resources = Store.objects.live().extra(
             where=[
                 'pootle_store_store.pootle_path LIKE %s',
-                'pootle_store_store.pootle_path ' + sql_not_regex + ' %s',
-            ], params=[query_pootle_path, disabled_tps_regex]
+                'pootle_store_store.pootle_path ' + sql_regex + ' %s',
+            ], params=[query_pootle_path, user_tps_regex]
         ).select_related('translation_project__language')
     else:
         resources = Directory.objects.live().extra(
             where=[
                 'pootle_app_directory.pootle_path LIKE %s',
-                'pootle_app_directory.pootle_path ' + sql_not_regex + ' %s',
-            ], params=[query_pootle_path, disabled_tps_regex]
+                'pootle_app_directory.pootle_path ' + sql_regex + ' %s',
+            ], params=[query_pootle_path, user_tps_regex]
         ).select_related('parent')
 
     if not resources.exists():


Test patch:
Repository: translate/pootle

Issue:
Using dynamic choices on model causes migration issues
There are some dynamic choice on models - if a deployment env is set up differently to the env in which the last migration was created - it causes makemigrations to want to create a new migration.

See here:

https://github.com/translate/pootle/blob/master/pootle/apps/pootle_project/models.py#L159:L160

Generally choices shouldnt be dynamic in this way:

https://docs.djangoproject.com/en/dev/ref/models/fields/#choices

See also #3687



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/forms.py b/pootle/apps/pootle_app/forms.py
--- a/pootle/apps/pootle_app/forms.py
+++ b/pootle/apps/pootle_app/forms.py
@@ -16,6 +16,7 @@
 
 from pootle_language.models import Language
 from pootle_project.models import Project, RESERVED_PROJECT_CODES
+from pootle_store.filetypes import filetype_choices
 from pootle_store.models import Store
 
 
@@ -76,6 +77,11 @@ def clean_localfiletype(self):
             value = self.cleaned_data.get('localfiletype', None)
             if not value:
                 value = self.instance.localfiletype
+            filetypes = [x[0] for x in filetype_choices]
+            if value not in filetypes:
+                raise forms.ValidationError(
+                    "Unrecognised project filetype: %s" % value
+                )
             return value
 
         def clean_treestyle(self):
diff --git a/pootle/apps/pootle_project/migrations/0002_remove_dynamic_model_choices_localfiletype.py b/pootle/apps/pootle_project/migrations/0002_remove_dynamic_model_choices_localfiletype.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/pootle_project/migrations/0002_remove_dynamic_model_choices_localfiletype.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pootle_project', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='project',
+            name='localfiletype',
+            field=models.CharField(default=b'po', max_length=50, verbose_name='File Type'),
+            preserve_default=True,
+        ),
+    ]
diff --git a/pootle/apps/pootle_project/models.py b/pootle/apps/pootle_project/models.py
--- a/pootle/apps/pootle_project/models.py
+++ b/pootle/apps/pootle_project/models.py
@@ -157,7 +157,7 @@ class Project(models.Model, CachedTreeItem, ProjectURLMixin):
             verbose_name=_('Quality Checks'))
 
     localfiletype = models.CharField(max_length=50, default="po",
-            choices=filetype_choices, verbose_name=_('File Type'))
+                                     verbose_name=_('File Type'))
 
     treestyle_choices = (
             # TODO: check that the None is stored and handled correctly


Test patch:
Repository: translate/pootle

Issue:
update_tmserver shouldn't use disabled projects
`update_tmserver` command shouldn't use disabled projects unless they are specifically requested.

By default we shouldn't use them.  If the user supplies a command line option or uses a specific project name they we should load that project into the TM.



Discussion:
Probably it is better to add a `--include-disabled-projects` instead of just dropping the functionality.


Resolving patch:
diff --git a/pootle/apps/pootle_app/management/commands/update_tmserver.py b/pootle/apps/pootle_app/management/commands/update_tmserver.py
--- a/pootle/apps/pootle_app/management/commands/update_tmserver.py
+++ b/pootle/apps/pootle_app/management/commands/update_tmserver.py
@@ -32,6 +32,7 @@ class DBParser(object):
     def __init__(self, *args, **kwargs):
         self.stdout = kwargs.pop('stdout')
         self.INDEX_NAME = kwargs.pop('index', None)
+        self.exclude_disabled_projects = not kwargs.pop('disabled_projects')
 
     def get_units(self, *filenames):
         """Gets the units to import and its total count."""
@@ -44,7 +45,14 @@ def get_units(self, *filenames):
                 'store',
                 'store__translation_project__project',
                 'store__translation_project__language'
-            ).values(
+            )
+
+        if self.exclude_disabled_projects:
+            units_qs = units_qs.exclude(
+                store__translation_project__project__disabled=True
+            )
+
+        units_qs = units_qs.values(
                 'id',
                 'revision',
                 'source_f',
@@ -169,6 +177,12 @@ class Command(BaseCommand):
                     dest='dry_run',
                     default=False,
                     help='Report the number of translations to index and quit'),
+        # Local TM specific options.
+        make_option('--include-disabled-projects',
+                    action='store_true',
+                    dest='disabled_projects',
+                    default=False,
+                    help='Add translations from disabled projects'),
         # External TM specific options.
         make_option('--tm',
                     action='store',
@@ -256,7 +270,8 @@ def _initialize(self, *args, **options):
             raise CommandError('You cannot add translations from database to '
                                'an external TM.')
         else:
-            self.parser = DBParser(stdout=self.stdout, index=self.INDEX_NAME)
+            self.parser = DBParser(stdout=self.stdout, index=self.INDEX_NAME,
+                                   disabled_projects=options.pop('disabled_projects'))
 
     def _set_latest_indexed_revision(self, **options):
         self.last_indexed_revision = -1


Test patch:
Repository: translate/pootle

Issue:
Broken l10n in timeline
Issue #3824 requested a comment for this unit:

```
templates/editor/units/xhr_timeline.html:30
    check
```

> This one could use some translator comment, what is it used for?

Inspection of [that code](https://github.com/translate/pootle/blob/7907606c/pootle/templates/editor/units/xhr_timeline.html#L37) shows some really damaged localisation which appears in the timeline we guess as:

'Muted Placeholder check'

I.e. we've got a constructed concatenation that many languages can't translate.  And we're not 100% sure where the first part is localisable anyway.



Discussion:
The other part comes [from the view](https://github.com/translate/pootle/blob/c4d794bbfe7568b05849ccffb2f32517f9c5ac43/pootle/apps/pootle_store/views.py#L626-L627), which is as braindead as the template construct.


Resolving patch:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -636,11 +636,9 @@ def timeline(request, unit):
                 entry.update({
                     'check_name': check_name,
                     'check_display_name': check_names[check_name],
-                    'checks_url': reverse('pootle-checks-descriptions'),
-                    'action': {
-                                SubmissionTypes.MUTE_CHECK: 'Muted',
-                                SubmissionTypes.UNMUTE_CHECK: 'Unmuted'
-                              }.get(item.type, '')
+                    'checks_url': u''.join([
+                        reverse('pootle-checks-descriptions'), '#', check_name,
+                    ]),
                 })
             else:
                 entry['new_value'] = to_python(item.new_value)


Test patch:
Repository: translate/pootle

Issue:
Change icons for v-folders
To better distinguish virtual folders (or "goals") from regular folders, let's use the following icon:

![icon](https://cloud.githubusercontent.com/assets/1728158/11135103/d40161b6-8956-11e5-9a82-cc0ff4e22527.png)

Preview:

![vfolder_icons_preview](https://cloud.githubusercontent.com/assets/1728158/11135109/d8f3f8a0-8956-11e5-8a8b-7de338c3b7d6.png)



Discussion:

Resolving patch:
diff --git a/pootle/core/browser.py b/pootle/core/browser.py
--- a/pootle/core/browser.py
+++ b/pootle/core/browser.py
@@ -199,7 +199,7 @@ def make_vfolder_treeitem(vfolder_treeitem):
         'code': vfolder_treeitem.code,
         'priority': vfolder_treeitem.vfolder.priority,
         'is_grayed': not vfolder_treeitem.is_visible,
-        'icon': 'folder',
+        'icon': 'vfolder',
     }
 
 


Test patch:
Repository: translate/pootle

Issue:
A user can't download the terminology files from a Terminology project
I know we switched off download of `pootle-terminology.po` files as they cause a number of problem mostly because there is no associated file.  But this seems to have extended to also include the 'Terminology' project.

I can't recall if at the time we decided it was an OK compromise.  But now with @phlax's work which means we can have non-existent files we should likely be able to bring this back. At the very least we shouldn't be disabling download of files within Terminology ad only restrict `pootle-terminology.po`



Discussion:
By reading #3937 it doesn't look like this is intended — the referenced commit is entirely prohibiting (or at least hiding) downloads from the terminology project.

The work on `pootle_fs` to break away from the need for a file on the filesystem should make it possible to revert the #3937 fix and get downloads of terminology back again.

@dwaynebailey ive re-enabled this and done a really quick ttw test to check it works ok.

Is there some special handling necessary for pootle-terminology.po?

I cant actually find that file in the default build


Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -62,12 +62,11 @@ def browse(request, translation_project, dir_path, filename=None):
 
     # TODO improve plugin logic
     if "import_export" in settings.INSTALLED_APPS:
-        if not translation_project.is_terminology_project:
-            if request.user.is_authenticated():
-                if check_permission('translate', request):
-                    ctx.update(handle_upload_form(request, project))
-                ctx.update({'display_download': True,
-                            'has_sidebar': True})
+        if request.user.is_authenticated():
+            if check_permission('translate', request):
+                ctx.update(handle_upload_form(request, project))
+            ctx.update({'display_download': True,
+                        'has_sidebar': True})
 
     stats = request.resource_obj.get_stats()
 


Test patch:
Repository: translate/pootle

Issue:
Uploads perform poorly when some non localisation error is picked up
On a Mac you could have stray `.DS_Store` files.  Also when using POedit it will create a `.mo` files whenever you save the `.po` file.

Errors such as `Unknown filetype (en_ZA/firefox/browser/chrome/overrides/.DS_Store)` are reported in this case.

Whenever such an error occurs then the upload reports the error and fails to complete.  We should I think ignore errors unrelated to the translations files we are uploading.  And at least execute correctly for those where there are no errors.



Discussion:

Resolving patch:
diff --git a/pootle/apps/import_export/views.py b/pootle/apps/import_export/views.py
--- a/pootle/apps/import_export/views.py
+++ b/pootle/apps/import_export/views.py
@@ -64,10 +64,12 @@ def export(request):
         return download(f.getvalue(), "%s.zip" % (prefix), "application/zip")
 
 
-def handle_upload_form(request):
+def handle_upload_form(request, project):
     """Process the upload form."""
     if request.method == "POST" and "file" in request.FILES:
         upload_form = UploadForm(request.POST, request.FILES)
+        project_filetypes = [project.localfiletype,
+                             project.get_template_filetype()]
 
         if upload_form.is_valid():
             django_file = request.FILES["file"]
@@ -78,6 +80,9 @@ def handle_upload_form(request):
                             if path.endswith("/"):
                                 # is a directory
                                 continue
+                            ext = os.path.splitext(path)[1].strip(".")
+                            if ext not in project_filetypes:
+                                continue
                             with zf.open(path, "r") as f:
                                 import_file(f, user=request.user)
                 else:
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -131,7 +131,7 @@ def browse(request, translation_project, dir_path, filename=None):
     if "import_export" in settings.INSTALLED_APPS and request.user.is_authenticated():
         from import_export.views import handle_upload_form
 
-        ctx.update(handle_upload_form(request))
+        ctx.update(handle_upload_form(request, project))
 
         has_download = (not translation_project.is_terminology_project and
                         (check_permission('translate', request) or


Test patch:
Repository: translate/pootle

Issue:
Missing migration for staticpages.AbstractPage
When I do `pootle makemigrations`, Django detects an inconsistency and generates new migration to add fields `url` to `staticpages.legalpage` and `staticpages.staticpage`. Not sure if it's a missing migration, or a field which doesn't have to be there in the first place.

Thanks in advance for fixing this one way or another.



Discussion:
Thank you for reporting.

Can you please provide some additional information, like the version you were upgrading from and the exact commands you run and the output?

@unho the error occured becuase of 08ac39503a98c6b8bcd76f36565e7182e09be576 which changed the `help_text`

@unho 

```
$ git rev-parse HEAD
ed7c9f289ef753d33393d5bcf09038e9100fe876

$ pootle migrate
  ...
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

$ pootle makemigrations
...
Migrations for 'staticpages':
  0002_auto_20151123_1046.py:
    - Alter field url on legalpage
    - Alter field url on staticpage
```

@dwaynebailey: True, turned out it's not a big deal :)

@unho yip, luckily.  I'm busy testing something in .travis that will catch these in future.


Resolving patch:
diff --git a/pootle/apps/staticpages/migrations/0002_change_url_field_help_text.py b/pootle/apps/staticpages/migrations/0002_change_url_field_help_text.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/staticpages/migrations/0002_change_url_field_help_text.py
@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('staticpages', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='legalpage',
+            name='url',
+            field=models.URLField(help_text='If set, this page will redirect to this URL', verbose_name='Redirect to URL', blank=True),
+            preserve_default=True,
+        ),
+        migrations.AlterField(
+            model_name='staticpage',
+            name='url',
+            field=models.URLField(help_text='If set, this page will redirect to this URL', verbose_name='Redirect to URL', blank=True),
+            preserve_default=True,
+        ),
+    ]


Test patch:
Repository: translate/pootle

Issue:
Enable sorting by contribution in contributor command
Currently contributors are sorted in alphabetical order.  This is great for crediting.  But it would be more helpful to allow sorting by contribution in cases where you want to use to list to make other decisions around the amount of contribution.

Thus add `--sort-by-contribution` and `--sort-by-name` options.



Discussion:
What about using just `--sort-by` and passing a value to it like in `--sort-by=contributions` or `--sort-by=name`?

@unho yip that works and allows extension if needed.


Resolving patch:
diff --git a/pootle/apps/pootle_app/management/commands/contributors.py b/pootle/apps/pootle_app/management/commands/contributors.py
--- a/pootle/apps/pootle_app/management/commands/contributors.py
+++ b/pootle/apps/pootle_app/management/commands/contributors.py
@@ -32,6 +32,15 @@ class Command(PootleCommand):
             dest="revision",
             help="Only count contributions newer than this revision",
         ),
+        make_option(
+            "--sort-by",
+            type="choice",
+            default="name",
+            choices=["name", "contributions"],
+            dest="sort_by",
+            help="Sort by specified item. Accepts name and contributions. "
+                 "Default: %default",
+        ),
     )
 
     help = "Print a list of contributors."
@@ -58,17 +67,25 @@ def handle_all(self, **options):
         for v in units.values("submitted_by"):
             contribs.update((v["submitted_by"], ))
 
-        self.list_contributions(contribs)
+        self.list_contributions(contribs, options["sort_by"])
+
+    def list_contributions(self, contribs, sort_by):
+        if sort_by == "name":
+            contributions = contribs.items()
+        else:
+            contributions = contribs.most_common()
 
-    def list_contributions(self, contribs):
         out = []
-        for id, count in contribs.items():
+        for id, count in contributions:
             user = User.objects.get(id=id)
             name = user.display_name
             if user.email:
                 name += " <%s>" % (user.email)
             out.append("%s (%i contributions)" % (name, count))
 
-        # Sort users alphabetically
-        for line in sorted(out):
+        if sort_by == "name":
+            # Sort users alphabetically
+            out = sorted(out)
+
+        for line in out:
             self.stdout.write(line)


Test patch:
Repository: translate/pootle

Issue:
PootleCommand handles obsolete translation projects
`update_stores` command can make obsolete TP directory many times. 
It doesn't affect us much except we get extra unnecessary log messages like this:

```
set(['get_last_updated', 'get_checks', 'get_mtime', 'get_suggestion_count', 'get_last_action', 'get_wordcount_stats']) deleted from /uk/android_announcements_evernote/ cache
```

which tells us that cache for TP has been deleted.

I think any `PootleCommand` shouldn't handle TP with obsolete directories. `update_stores` command will resurrect TP directory when it's added.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/management/commands/__init__.py b/pootle/apps/pootle_app/management/commands/__init__.py
--- a/pootle/apps/pootle_app/management/commands/__init__.py
+++ b/pootle/apps/pootle_app/management/commands/__init__.py
@@ -136,7 +136,7 @@ def handle_all(self, **options):
             project_query = project_query.filter(code__in=self.projects)
 
         for project in project_query.iterator():
-            tp_query = project.translationproject_set \
+            tp_query = project.translationproject_set.live() \
                               .order_by('language__code')
 
             if self.languages:


Test patch:
Repository: translate/pootle

Issue:
select_related breaks order_by in UnitManager.get_queryset
The default manager for Unit does select_related - here:

https://github.com/translate/pootle/blob/master/pootle/apps/pootle_store/models.py#L223:L224

This seems to break order_by on the model, which is used by the units property on Store, here:

https://github.com/translate/pootle/blob/master/pootle/apps/pootle_store/models.py#L1400

The problem only occurs when you try to access Store.units by index, ie:

``` python
for i in range(0, store.units.count()):
    print store.units[i]
```

So the following work correctly as they iterate the units

``` python
for unit in store.units:
    print unit.index

for index store.units.values_list("index", flat=True):
    print index
```

This seems to be a Django bug - but im not sure, and couldnt find anything that had been already reported



Discussion:
Is this perhaps a side-effect of #3837?


Resolving patch:
diff --git a/pootle/apps/pootle_store/models.py b/pootle/apps/pootle_store/models.py
--- a/pootle/apps/pootle_store/models.py
+++ b/pootle/apps/pootle_store/models.py
@@ -234,14 +234,6 @@ def stringcount(string):
 
 class UnitManager(models.Manager):
 
-    def get_queryset(self):
-        """Mimics `select_related(depth=1)` behavior. Pending review."""
-        return (
-            super(UnitManager, self).get_queryset().select_related(
-                'store', 'submitted_by', 'commented_by', 'reviewed_by',
-            )
-        )
-
     def live(self):
         """Filters non-obsolete units."""
         return self.filter(state__gt=OBSOLETE)
@@ -1442,8 +1434,7 @@ def units(self):
         if hasattr(self, '_units'):
             return self._units
 
-        return self.unit_set.filter(state__gt=OBSOLETE).order_by('index') \
-                            .select_related('store__translation_project')
+        return self.unit_set.filter(state__gt=OBSOLETE).order_by('index')
 
     @units.setter
     def units(self, value):


Test patch: