internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: translate/pootle

Issue:
Add an ability to show/hide disabled projects
Now  that we only disable projects, not delete them, the list of disabled projects can grow and distract from the active projects. Let's add a toggle that would allow to show /hide disabled projects (and have them hidden by default). The toggle would say either "Hide disabled" or "Show disabled" and would only appear for admins and only when there's at least one disabled project (or translation project):

![screenclip 4](https://cloud.githubusercontent.com/assets/1728158/14097400/4149e1c8-f525-11e5-9563-29c7813e44b4.png)

By default on every page load the disabled projects would be hidden, and the link would say "Show disabled".

Implementation-wise, this can be done in CSS by toggling a global class on a table that would hide or show the disabled rows. This, however, will require the change in how we highlight odd/even rows. See e.g. http://stackoverflow.com/questions/6629576/make-css-nth-child-only-affect-visible



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_project/views.py b/pootle/apps/pootle_project/views.py
--- a/pootle/apps/pootle_project/views.py
+++ b/pootle/apps/pootle_project/views.py
@@ -353,7 +353,7 @@ class ProjectsBrowseView(ProjectsMixin, PootleBrowseView):
         'name', 'progress', 'total', 'need-translation',
         'suggestions', 'critical', 'last-updated', 'activity']
 
-    @property
+    @cached_property
     def items(self):
         items = [
             make_project_list_item(project)
diff --git a/pootle/core/views.py b/pootle/core/views.py
--- a/pootle/core/views.py
+++ b/pootle/core/views.py
@@ -581,6 +581,10 @@ def sidebar_announcements(self):
             self.request,
             (self.object, ))
 
+    @property
+    def disabled_items(self):
+        return filter(lambda item: item['is_disabled'], self.items)
+
     @property
     def table(self):
         if self.table_id and self.table_fields and self.items:
@@ -588,7 +592,9 @@ def table(self):
                 'id': self.table_id,
                 'fields': self.table_fields,
                 'headings': get_table_headings(self.table_fields),
-                'items': self.items}
+                'items': self.items,
+                'disabled_items': self.disabled_items,
+            }
 
     def get(self, *args, **kwargs):
         response = super(PootleBrowseView, self).get(*args, **kwargs)


Test patch:
Repository: translate/pootle

Issue:
Make `pootle webpack` not require system checks
`pootle webpack` fails if eg the db is not set up/correctly. It would be helpful if it didnt



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_app/management/commands/webpack.py b/pootle/apps/pootle_app/management/commands/webpack.py
--- a/pootle/apps/pootle_app/management/commands/webpack.py
+++ b/pootle/apps/pootle_app/management/commands/webpack.py
@@ -19,6 +19,7 @@
 
 class Command(BaseCommand):
     help = 'Builds and bundles static assets using webpack'
+    requires_system_checks = False
 
     def add_arguments(self, parser):
         parser.add_argument(


Test patch:
Repository: translate/pootle

Issue:
Disable downloading of  non-PO files
This issue https://github.com/translate/pootle/issues/4790 requests upload functionality for downloaded non-PO formats.
But XLIFF files are downloaded in a broken format with changed unit ids. It'd be safer to disable download functionality for non-PO files.



Discussion:
Sometimes makes sense to allow download despite upload is disabled, for example to allow people to get a backup for their translations.

But at least upload must be disabled for formats like XLIFF, where upload still doesn't properly work.

yep, i have a feeling that some translators might export to other formats to use the translations in other tools - not sure. Im wondering if adding a message that upload isnt (currently) supported would be better than disabling altogether

@phlax I think the issue would be that we aren't exporting a format that we can safely import.  Disabling upload may be a good first step.  I'm not sure how we do with other formats.  If the download was even vaguely similar to what we got out of `sync_stores` then fine.  I think serializer helps here.  My gut disable upload is minimal best for now.  Disabling all download and upload would prevent export but would prevent the idea that you can work offline in this format.

Currently when we get such message after uploading non-PO files.
![image](https://cloud.githubusercontent.com/assets/682845/16344791/1d989320-3a47-11e6-9255-a5cafc5b470f.png)
It's very likely to minimal best for now (according @dwaynebailey). If we just want to disable upload I'd do nothing.


Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -243,7 +243,8 @@ def get_upload_widget(self, project):
             and self.request.user.is_authenticated()
             and check_permission('translate', self.request))
         if has_upload:
-            ctx.update(handle_upload_form(self.request, project))
+            if self.project.localfiletype == 'po':
+                ctx.update(handle_upload_form(self.request, project))
             ctx.update(
                 {'display_download': True,
                  'has_sidebar': True})


Test patch:
Repository: translate/pootle

Issue:
User contributions for their top language don't work
User contributions for their top language always displays _No contributions in the last 30 days_ in their profile page.

This is likely related to some recent changes on how contributions are retrieved.



Discussion:
I believe the problem is that `User.top_scorers` doesn't return a user anymore, but a dictionary with annotated data.


Resolving patch:
diff --git a/pootle/apps/accounts/models.py b/pootle/apps/accounts/models.py
--- a/pootle/apps/accounts/models.py
+++ b/pootle/apps/accounts/models.py
@@ -424,8 +424,8 @@ def top_language(self, days=30):
         if language is not None:
             language_scorers = self.top_scorers(language=language.code,
                                                 days=days, limit=None)
-            for index, user in enumerate(language_scorers, start=1):
-                if user == self:
+            for index, user_score in enumerate(language_scorers, start=1):
+                if user_score['user'] == self:
                     position = index
                     break
 


Test patch:
Repository: translate/pootle

Issue:
Exception in terminology management view
When visiting https://mozilla.locamotion.org/eu/firefox/terminology/ the following exception is thrown:

`'SortedRelatedManager' object does not support indexing`



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_terminology/views.py b/pootle/apps/pootle_terminology/views.py
--- a/pootle/apps/pootle_terminology/views.py
+++ b/pootle/apps/pootle_terminology/views.py
@@ -27,7 +27,7 @@ def get_terminology_filename(translation_project):
 
     return (
         'pootle-terminology.%s'
-        % translation_project.project.filetypes[0].extension)
+        % translation_project.project.filetypes.first().extension)
 
 
 def manage_store(request, ctx, language, term_store):


Test patch:
Repository: translate/pootle

Issue:
Files dropdown keeps references to empty directories
In https://mozilla.locamotion.org/templates/firefox/toolkit/chrome/mozapps/plugins/ there were files that have been removed a while ago, but you can still select this empty directory in the dropdown. This **plugins** directory is not displayed in the statistics table at https://mozilla.locamotion.org/templates/firefox/toolkit/chrome/mozapps/



Discussion:
The project resources cache is only invalidated after a DB directory deletion via the `post_delete` signal (same for stores); obsoleting it won't invalidate it. I believe in-DB directories are obsoleted when the directories are removed from the FS, right @ta2-1? If that's the case, the cache should also be invalidated in such a scenario.


Resolving patch:
diff --git a/pootle/apps/pootle_project/models.py b/pootle/apps/pootle_project/models.py
--- a/pootle/apps/pootle_project/models.py
+++ b/pootle/apps/pootle_project/models.py
@@ -583,8 +583,13 @@ def invalidate_resources_cache(**kwargs):
         return
 
     # Don't invalidate if the save didn't create new objects
-    if (('created' in kwargs and 'raw' in kwargs) and
-        (not kwargs['created'] or kwargs['raw'])):
+    no_new_objects = (
+        ('created' in kwargs
+         and 'raw' in kwargs)
+        and (not kwargs['created']
+             or kwargs['raw']))
+
+    if no_new_objects and instance.parent.get_children():
         return
 
     proj_code = split_pootle_path(instance.pootle_path)[1]


Test patch:
Repository: translate/pootle

Issue:
Update/sync update obsolete units every time with --force --overwrite
While testing recent changes to the Store.sync code i noticed that on every sync units that were obsolete were being shown as `deleted`.

On further testing it appears that Store.update does something similar with the same units, effectively generating noop submission records.

In the case of Store.update it appears that this has been the case for some time.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/syncer.py b/pootle/apps/pootle_store/syncer.py
--- a/pootle/apps/pootle_store/syncer.py
+++ b/pootle/apps/pootle_store/syncer.py
@@ -205,10 +205,10 @@ def get_new_units(self, old_ids, new_ids):
              in new_ids - old_ids])
 
     def get_units_to_obsolete(self, old_ids, new_ids):
-        return (
-            self.disk_store.findid(uid)
-            for uid
-            in old_ids - new_ids)
+        for uid in old_ids - new_ids:
+            unit = self.disk_store.findid(uid)
+            if unit and not unit.isobsolete():
+                yield unit
 
     def obsolete_unit(self, unit, conservative):
         deleted = not unit.istranslated()
diff --git a/pootle/apps/pootle_store/updater.py b/pootle/apps/pootle_store/updater.py
--- a/pootle/apps/pootle_store/updater.py
+++ b/pootle/apps/pootle_store/updater.py
@@ -123,7 +123,8 @@ def should_create_suggestion(self):
     def should_update_index(self):
         return (
             self.update.change_indices
-            and self.uid in self.update.indices)
+            and self.uid in self.update.indices
+            and self.unit.index != self.update.get_index(self.uid))
 
     @property
     def should_update_target(self):


Test patch:
diff --git a/tests/models/store.py b/tests/models/store.py
--- a/tests/models/store.py
+++ b/tests/models/store.py
@@ -1369,7 +1369,9 @@ def _test_get_obsolete(results, syncer, old_ids, new_ids):
     assert list(results) == list(
         syncer.disk_store.findid(uid)
         for uid
-        in old_ids - new_ids)
+        in old_ids - new_ids
+        if (syncer.disk_store.findid(uid)
+            and not syncer.disk_store.findid(uid).isobsolete()))
 
 
 @pytest.mark.django_db
Repository: translate/pootle

Issue:
Ensure tests can be run with `--reuse-db`
When iterating over a test that require DB access (or a few of them), currently a site-wide setup is made which in such scenario ends up being relatively time-consuming and tedious.

Ideally one could use [pytest-django's `--reuse-db` flag](http://pytest-django.readthedocs.org/en/latest/database.html#reuse-db-reuse-the-testing-database-between-test-runs) to considerably reduce setup time on test iterations, however at the current state of things such feature cannot be used due to the way the Pootle test DB environment is setup.

Let's try to fix that so we can benefit from `--reuse-db`.



Discussion:

Resolving patch:
diff --git a/pytest_pootle/plugin.py b/pytest_pootle/plugin.py
--- a/pytest_pootle/plugin.py
+++ b/pytest_pootle/plugin.py
@@ -85,7 +85,7 @@ def tests_use_migration(request, tests_use_db):
 @pytest.fixture(autouse=True, scope='session')
 def setup_db_if_needed(request, tests_use_db):
     """Sets up the site DB only if tests requested to use the DB (autouse)."""
-    if tests_use_db:
+    if tests_use_db and not request.config.getvalue('reuse_db'):
         return request.getfuncargvalue('post_db_setup')
 
 


Test patch:
Repository: translate/pootle

Issue:
Severe performance degradation of sync_stores
Earlier today we updated production translation server (merged 'Raw font' PR branch with master and switched to master). So this release includes recent changes related to optimizations. Immediately after that our sync cycle time increased from typical 18-20 minutes to 1.5 hours. I had to write a tool to analyze and compare our logs from past sync cycles and extract timing information. Here's the output:

![screenclip](https://cloud.githubusercontent.com/assets/1728158/18117500/035f5506-6f04-11e6-98db-5303e76872f7.png)

Some comments on the screenshot. It compares our main steps of sync cycle between four logs (vertical columns). The first log is from a morning run, some time before the release. The second one is soon after the release, and the last two are most recent ones. As one might see, the `pull-ts` step is the culprit. It went from mere 3 minutes up to more than an hour. During this step all we do is run `manage.py sync_stores --skip-missing --project=<project_id>` and wait for it to finish.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/syncer.py b/pootle/apps/pootle_store/syncer.py
--- a/pootle/apps/pootle_store/syncer.py
+++ b/pootle/apps/pootle_store/syncer.py
@@ -256,7 +256,8 @@ def create_store_file(self, last_revision, user):
     def update_newer(self, last_revision):
         return (
             not self.store.file.exists()
-            or (last_revision >= self.store.last_sync_revision))
+            or last_revision > self.store.last_sync_revision
+        )
 
     @cached_property
     def dbid_index(self):


Test patch:
Repository: translate/pootle

Issue:
Wrong TP relative store path in breadcrumbs on translate pages
![image](https://cloud.githubusercontent.com/assets/682845/18382056/123f3db2-7688-11e6-969c-93dbb2369557.png)

I think we need to clarify this property in the Store model. 

``` python
@cached_property
    def path(self):
        """Returns just the path part omitting language and project codes.
        If the `pootle_path` of a :cls:`Store` object `store` is
        `/af/project/dir1/dir2/file.po`, `store.path` will return
        `dir1/dir2/file.po`.
        """
        return self.pootle_path.split(u'/', 2)[-1]
```



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/models.py b/pootle/apps/pootle_store/models.py
--- a/pootle/apps/pootle_store/models.py
+++ b/pootle/apps/pootle_store/models.py
@@ -41,7 +41,8 @@
 from pootle.core.search import SearchBroker
 from pootle.core.storage import PootleFileSystemStorage
 from pootle.core.url_helpers import (
-    get_all_pootle_paths, get_editor_filter, split_pootle_path)
+    get_all_pootle_paths, get_editor_filter, split_pootle_path,
+    to_tp_relative_path)
 from pootle.core.utils import dateformat
 from pootle.core.utils.aggregate import max_column
 from pootle.core.utils.timezone import datetime_min, make_aware
@@ -1209,7 +1210,7 @@ def path(self):
         `/af/project/dir1/dir2/file.po`, `store.path` will return
         `dir1/dir2/file.po`.
         """
-        return self.pootle_path.split(u'/', 2)[-1]
+        return to_tp_relative_path(self.pootle_path)
 
     def __init__(self, *args, **kwargs):
         super(Store, self).__init__(*args, **kwargs)


Test patch:
diff --git a/tests/models/store.py b/tests/models/store.py
--- a/tests/models/store.py
+++ b/tests/models/store.py
@@ -28,6 +28,7 @@
     config, format_classes, format_diffs, formats)
 from pootle.core.models import Revision
 from pootle.core.delegate import deserializers, serializers
+from pootle.core.url_helpers import to_tp_relative_path
 from pootle.core.plugin import provider
 from pootle.core.serializers import Serializer, Deserializer
 from pootle_app.models import Directory
@@ -1432,3 +1433,8 @@ def test_store_syncer_new_units(dummy_store_syncer_units, tp0):
         expected["old_ids"], expected["new_ids"])
     _test_get_new(
         results, syncer, expected["old_ids"], expected["new_ids"])
+
+
+@pytest.mark.django_db
+def test_store_path(store0):
+    assert store0.path == to_tp_relative_path(store0.pootle_path)
Repository: translate/pootle

Issue:
Checks fail to detect non-installed Redis
This error was hit when setting up Pootle in a dev environment. Redis was not installed.

``` pytb
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line()
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django/core/management/base.py", line 398, in execute
    self.check()
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django/core/management/base.py", line 426, in check
    include_deployment_checks=include_deployment_checks,
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 75, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/home/charles/codes/translatehouse/pootle/pootle/checks.py", line 437, in check_revision
    revision = Revision.get()
  File "/home/charles/codes/translatehouse/pootle/pootle/core/models/revision.py", line 45, in get
    return cache.get(cls.CACHE_KEY)
  File "/home/charles/.virtualenvwrapper/envs/pootle-dev/local/lib/python2.7/site-packages/django_redis/cache.py", line 39, in _decorator
    raise e.parent
redis.exceptions.ConnectionError: Error 111 connecting to 127.0.0.1:6379. Connection refused.
```



Discussion:

Resolving patch:
diff --git a/pootle/checks.py b/pootle/checks.py
--- a/pootle/checks.py
+++ b/pootle/checks.py
@@ -430,11 +430,16 @@ def check_email_server_is_alive(app_configs=None, **kwargs):
 
 @checks.register('data')
 def check_revision(app_configs=None, **kwargs):
+    from redis.exceptions import ConnectionError
+
     from pootle.core.models import Revision
     from pootle_store.models import Unit
 
     errors = []
-    revision = Revision.get()
+    try:
+        revision = Revision.get()
+    except (ConnectionError):
+        return errors
     try:
         max_revision = Unit.max_revision()
     except (OperationalError, ProgrammingError):


Test patch:
Repository: translate/pootle

Issue:
Suggestion review breaks with edited units
Reported initially by @bellayet and we did the following test to replicate:
1. @bellayet submitted a Bengali suggestion to [an untranslated en_ZA units](https://mozilla.locamotion.org/en_ZA/firefox/translate/browser/chrome/browser/aboutSearchReset.dtd.po#unit=21737427&offset=0)
2. @dwaynebailey then used UI to alter the text (remove all Bengali and use correct English string) and submit a thank you

Results:
1. On accepting we moved forward
2. Previous row showed correct translation
3. The next unit is very similar to the suggestion unit.  TM showed Bengali as a suggestion, so suggestion made it into the TM
4. Stepping back to the unit
5. The translation was blank i.e. no translation i.e. exactly as it was pre suggestion acceptance
6. Looking at unit history
7. It looks like we accepted the suggestion, but then I'm also not sure if the details explanation also covers changes.  i.e. do we just print your suggestion or do we show the final result.  Or should we say we "Accepted you suggestion X with edits to Y with comment Z"



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_store/views.py b/pootle/apps/pootle_store/views.py
--- a/pootle/apps/pootle_store/views.py
+++ b/pootle/apps/pootle_store/views.py
@@ -556,6 +556,7 @@ def submit(request, unit, **kwargs_):
             json['checks'] = _get_critical_checks_snippet(request, unit)
 
         json['user_score'] = request.user.public_score
+        json['newtargets'] = [target for target in form.instance.target.strings]
 
         return JsonResponse(json)
 


Test patch:
diff --git a/tests/views/unit.py b/tests/views/unit.py
--- a/tests/views/unit.py
+++ b/tests/views/unit.py
@@ -92,6 +92,7 @@ def test_submit_with_suggestion_and_comment(client, request_users, settings):
 
         content = json.loads(response.content)
 
+        assert content['newtargets'] == [edited_target]
         assert content['user_score'] == response.wsgi_request.user.public_score
         assert content['checks'] is None
Repository: translate/pootle

Issue:
Static page editing save HTML outcome instead of raw text
To reproduce:

1. Server using Markdown for static pages and an existing static page written using Markdown
2. Minor edit the content of the static page (editing the title also seems to do this)
3. Saves fine
4. Edit the page again.  Static page text is now all in HTML. 
5. View static page on site.  Static page is now all garbled HTML


Discussion:

Resolving patch:
diff --git a/pootle/core/markup/fields.py b/pootle/core/markup/fields.py
--- a/pootle/core/markup/fields.py
+++ b/pootle/core/markup/fields.py
@@ -119,8 +119,11 @@ def get_prep_value(self, value):
 
         return value
 
+    def to_python(self, value):
+        return self.get_prep_value(value)
+
     def value_to_string(self, obj):
-        value = self._get_val_from_obj(obj)
+        value = self.value_from_object(obj)
         return self.get_prep_value(value)
 
     def formfield(self, **kwargs):


Test patch:
Repository: translate/pootle

Issue:
Append priority sorting for translating directories when they have vfolder
After https://github.com/translate/pootle/pull/5483 (or maybe some other previous code landed after b57d23b) we lost the ability to append `&sort=priority` to the links to translate critical/suggestions/incomplete for the regular directories that have virtual folders below them (in other words, directories that might have units with different priorities).


Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_translationproject/views.py b/pootle/apps/pootle_translationproject/views.py
--- a/pootle/apps/pootle_translationproject/views.py
+++ b/pootle/apps/pootle_translationproject/views.py
@@ -288,10 +288,10 @@ def items(self):
             vf_stores = stores.filter(
                 vfolders__isnull=False).exclude(parent=self.object)
             dirs_with_vfolders = set(
-                path.replace(self.object.pootle_path, "").split("/")[0]
-                for path
-                in vf_stores.values_list(
-                    "pootle_path", flat=True))
+                [path.replace(self.object.pootle_path, "").split("/")[0]
+                 for path
+                 in vf_stores.values_list(
+                     "pootle_path", flat=True)])
         directories = [
             make_directory_item(
                 child,
diff --git a/pootle/core/browser.py b/pootle/core/browser.py
--- a/pootle/core/browser.py
+++ b/pootle/core/browser.py
@@ -80,6 +80,7 @@ def get_table_headings(choices):
 def make_generic_item(path_obj, **kwargs):
     """Template variables for each row in the table."""
     return {
+        'sort': kwargs.get("sort"),
         'href': path_obj.get_absolute_url(),
         'href_translate': path_obj.get_translate_url(),
         'title': path_obj.name,


Test patch: