internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: translate/pootle

Issue:
Improve POOTLE_META_USERS documentation
I'm trying to document the `POOTLE_META_USERS` setting.  The current docs are unclear as to what the purpose of the setting is, probably because of the term "API users".  I'm not 100% sure what I would put in there and what it would do.  Its unclear from the commit 70f7d95e6faeb70bcebc0cba8ee3623a32cc88b8 and the issue evernote/pootle#234 what exactly this is about.

But based on https://translate.evernote.com/user/system/ and https://translate.evernote.com/user/nobody/ I'm guessing those are `POOTLE_META_USERS`.  In which case should the default not be ('system', 'nobody')?

@ta2-1 you landed that feature, could you provide some guidance?



Discussion:
Meta users are essentially special non-human accounts. 'System' is a meta-user which makes changes in the system when 'update_stores' command is executed. 'Nobody' is a user not associated with anyone particular. Such users have a simplified public profile rendering (no social buttons, for example, and no scores).

In the future, there will be more meta-users (each meta-user being tied to a specific plugin and representing some external service, like localization provider, that sends their translations into Pootle. There's no much reason to change this setting manually, I guess.

I'd say that `POOTLE_META_USERS` is an internal setting which will likely be deprecated at some point.


Resolving patch:
diff --git a/pootle/apps/accounts/models.py b/pootle/apps/accounts/models.py
--- a/pootle/apps/accounts/models.py
+++ b/pootle/apps/accounts/models.py
@@ -139,11 +139,8 @@ def twitter_url(self):
     @cached_property
     def is_meta(self):
         """Returns `True` if this is a special fake user."""
-        meta_users = UserManager.META_USERS
-        if settings.POOTLE_META_USERS:
-            meta_users += settings.POOTLE_META_USERS
-
-        return self.username in meta_users
+        return self.username in UserManager.META_USERS + \
+                                settings.POOTLE_META_USERS
 
     @cached_property
     def email_hash(self):


Test patch:
Repository: translate/pootle

Issue:
VFolders: Abort creation and warn if there is a name clash
This is quickly provide a workaround for #3809 so 2.7.0 can be quickly released. A more long term proper fix must be provided for #3809.



Discussion:

Resolving patch:
diff --git a/pootle/apps/virtualfolder/models.py b/pootle/apps/virtualfolder/models.py
--- a/pootle/apps/virtualfolder/models.py
+++ b/pootle/apps/virtualfolder/models.py
@@ -172,10 +172,17 @@ def save(self, *args, **kwargs):
         # For each store create all VirtualFolderTreeItem tree structure up to
         # its adjusted vfolder location.
         for store in vfolder_stores_set:
-            VirtualFolderTreeItem.objects.get_or_create(
-                directory=store.parent,
-                vfolder=self,
-            )
+            try:
+                VirtualFolderTreeItem.objects.get_or_create(
+                    directory=store.parent,
+                    vfolder=self,
+                )
+            except ValidationError:
+                # If there is some problem, e.g. a clash with a directory,
+                # delete the virtual folder and all its related items, and
+                # reraise the exception.
+                self.delete()
+                raise
 
         # Get the set of projects whose resources cache must be invalidated.
         # This includes the projects the projects it was previously related to
@@ -355,6 +362,9 @@ def save(self, *args, **kwargs):
             self.directory.pootle_path
         )
 
+        # Force validation of fields.
+        self.clean_fields()
+
         # Trigger the creation of the whole parent tree up to the vfolder
         # adjusted location.
         if self.directory.pootle_path.count('/') > self.vfolder.location.count('/'):
@@ -381,6 +391,14 @@ def delete(self, *args, **kwargs):
 
         super(VirtualFolderTreeItem, self).delete(*args, **kwargs)
 
+    def clean_fields(self):
+        """Validate virtual folder tree item fields."""
+        if Directory.objects.filter(pootle_path=self.pootle_path).exists():
+            msg = (u"Problem adding virtual folder '%s' with location '%s': "
+                   u"VirtualFolderTreeItem clashes with Directory %s" %
+                   (self.vfolder.name, self.vfolder.location, self.pootle_path))
+            raise ValidationError(msg)
+
     def get_translate_url(self, **kwargs):
         lang, proj, dp, fn = split_pootle_path(self.pootle_path)
         return u''.join([


Test patch:
Repository: translate/pootle

Issue:
POT files counted into Need Translation
In statistics, string from POT files appears as "Need Translation" and therefore breaks stats about translation completeness

Pootle 2.6.0-rc1 (Django 1.12.0-rc1, Translate Toolkit 1.6.5)

POT files counted into Need Translation
In statistics, string from POT files appears as "Need Translation" and therefore breaks stats about translation completeness

Pootle 2.6.0-rc1 (Django 1.12.0-rc1, Translate Toolkit 1.6.5)



Discussion:
@unho Can you take a look at this while I'm away please?

I think all that may be required is checking whether the unit comes from a pot file when logging its stats.

Replicated. And it also adds up all the total counts.

It could be easier to replicate if people talked about "templates" TP.

`pootle-terminology.po` should also not be counted.

@unho Can you take a look at this while I'm away please?

I think all that may be required is checking whether the unit comes from a pot file when logging its stats.

Replicated. And it also adds up all the total counts.

It could be easier to replicate if people talked about "templates" TP.

`pootle-terminology.po` should also not be counted.


Resolving patch:
diff --git a/pootle/apps/pootle_store/models.py b/pootle/apps/pootle_store/models.py
--- a/pootle/apps/pootle_store/models.py
+++ b/pootle/apps/pootle_store/models.py
@@ -2162,9 +2162,14 @@ def _get_wordcount_stats(self):
             'translated': 0,
             'fuzzy': 0
         }
+
+        # only count files with an ext matching the project ext
+        proj_ext = "." + self.translation_project.project.localfiletype
+
         # XXX: `order_by()` here is important as it removes the default
         # ordering for units. See #3897 for reference.
-        res = self.units.order_by().values('state') \
+        res = self.units.filter(store__name__endswith=proj_ext) \
+                        .order_by().values('state') \
                         .annotate(wordcount=models.Sum('source_wordcount'))
         for item in res:
             ret['total'] += item['wordcount']


Test patch:
Repository: translate/pootle

Issue:
nplurals check not present after running ./manage.py regenerate_checks_descriptions
After I've run the `./manage.py regenerate_checks_descriptions` command the `nplurals` check is not present.  It is present in the StandardChecker so I'm not sure why it wouldn't be picked up.

nplurals check not present after running ./manage.py regenerate_checks_descriptions
After I've run the `./manage.py regenerate_checks_descriptions` command the `nplurals` check is not present.  It is present in the StandardChecker so I'm not sure why it wouldn't be picked up.



Discussion:
im working on this one...

@dwaynebailey the nplurals check is not present on `StandardChecker` - just on `StandardUnitsChecker`

im just trying to get my head around the difference

I have changed setup.py to generate descriptions for both StandardChecker and StandardUnitsChecker.

This works although i had to add "isreview" in to pootle_misc.checks.check_names - i'm figuring this is ok

@phlax this sound good.  Please add isreview to the excluded checks, search if isfuzzy in the checks.py to see where.

im working on this one...

@dwaynebailey the nplurals check is not present on `StandardChecker` - just on `StandardUnitsChecker`

im just trying to get my head around the difference

I have changed setup.py to generate descriptions for both StandardChecker and StandardUnitsChecker.

This works although i had to add "isreview" in to pootle_misc.checks.check_names - i'm figuring this is ok

@phlax this sound good.  Please add isreview to the excluded checks, search if isfuzzy in the checks.py to see where.


Resolving patch:
diff --git a/pootle/apps/pootle_misc/checks.py b/pootle/apps/pootle_misc/checks.py
--- a/pootle/apps/pootle_misc/checks.py
+++ b/pootle/apps/pootle_misc/checks.py
@@ -109,7 +109,7 @@
     'percent_brace_placeholders': _(u"Percent brace placeholders"),
 }
 
-excluded_filters = ['hassuggestion', 'spellcheck', 'isfuzzy', 'untranslated']
+excluded_filters = ['hassuggestion', 'spellcheck', 'isfuzzy', 'isreview', 'untranslated']
 
 # pre-compile all regexps
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -153,7 +153,8 @@ def run(self):
         import django
         import codecs
         from pootle.apps.pootle_misc.checks import check_names, excluded_filters
-        from translate.filters.checks import TeeChecker
+        from translate.filters.checks import (TeeChecker,
+                                              StandardChecker, StandardUnitChecker)
         try:
             from docutils.core import publish_parts
         except ImportError:
@@ -185,7 +186,9 @@ def get_check_description(name, filterfunc):
 
         # Get a checker with the Translate Toolkit checks. Note that filters
         # that are not used in Pootle are excluded.
-        fd = TeeChecker().getfilters(excludefilters=excluded_filters)
+        fd = TeeChecker(
+            checkerclasses=[StandardChecker, StandardUnitChecker]
+        ).getfilters(excludefilters=excluded_filters)
 
         docs = sorted(
             get_check_description(name, f) for name, f in fd.items()


Test patch:
Repository: translate/pootle

Issue:
User score isn't updated properly if a user rejects his own suggestion
If any user added suggestion then rejected his suggestion, user's score shouldn't changed.

We update score properly:
1. add `X` after user's suggestion;
2. add `Y` for a review of the suggestion;
3. subtract `X+Y` as penalty for the wrong suggestion;

But we try `2` and `3` within one request:

``` python
self.user.score += self.score_delta
self.user.save()
```

And we get `2` missed. As I see we are trying to update old score value. It's very likely that it happens because of per request transactions (So we can query for unchanged data even it's not a new async process as mentioned here #3974).



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_statistics/models.py b/pootle/apps/pootle_statistics/models.py
--- a/pootle/apps/pootle_statistics/models.py
+++ b/pootle/apps/pootle_statistics/models.py
@@ -11,6 +11,7 @@
 from django.contrib.auth import get_user_model
 from django.core.urlresolvers import reverse
 from django.db import models
+from django.db.models import F
 from django.template.defaultfilters import truncatechars
 from django.utils.functional import cached_property
 from django.utils.html import format_html
@@ -419,8 +420,10 @@ def save(self, *args, **kwargs):
 
         super(ScoreLog, self).save(*args, **kwargs)
 
-        self.user.score += self.score_delta
-        self.user.save()
+        User = get_user_model()
+        User.objects.filter(id=self.user.id).update(
+            score=F('score') + self.score_delta
+        )
         self.log()
 
     def log(self):


Test patch:
Repository: translate/pootle

Issue:
Visually differentiate disabled projects in the navigation dropdown
In the navigation dropdown, admins can see and select disabled projects (which is ok). The problem is that they have the same style as non-disabled ones.

Let's render disabled project as italic and with .5 opacity, to match their styling everywhere else in the UI.



Discussion:

Resolving patch:
diff --git a/pootle/apps/pootle_project/models.py b/pootle/apps/pootle_project/models.py
--- a/pootle/apps/pootle_project/models.py
+++ b/pootle/apps/pootle_project/models.py
@@ -62,9 +62,11 @@ def cached_dict(self, user):
         projects = cache.get(cache_key)
         if not projects:
             logging.debug('Cache miss for %s', cache_key)
+            projects_dict = self.for_user(user).order_by('fullname') \
+                                               .values('code', 'fullname',
+                                                       'disabled')
             projects = OrderedDict(
-                self.for_user(user).order_by('fullname')
-                                   .values_list('code', 'fullname')
+                (project.pop('code'), project) for project in projects_dict
             )
             cache.set(cache_key, projects, settings.POOTLE_CACHE_TIMEOUT)
 


Test patch:
Repository: translate/pootle

Issue:
Suggestions should always have a user
It was proposed in #3651 to enforce `Suggestion.user` to be never `NULL`.



Discussion:
We should assign suggestions without a user to `nobody`.

Already done in a 2.6 migration https://github.com/translate/pootle/blob/stable/2.6.0/pootle/apps/pootle_store/migrations/0022_assign_userless_suggestions_to_nobody.py This issue is about making sure a suggestion without user can be never created again.

If we did that in 2.6 why didn't we switch to `null=False` at the same time? :/

Everyone seemed to be okay to add the constraint, however [you asked not to do so](https://github.com/translate/pootle/issues/3651#issuecomment-86964569) until 2.7 was released.

Rats, I should have remembered that bug when we added vfolders - I said that when we thought there would be no more db changes :(

@ta2-1 @julen The fix can land now.


Resolving patch:
diff --git a/pootle/apps/pootle_store/migrations/0002_make_suggestion_user_not_null.py b/pootle/apps/pootle_store/migrations/0002_make_suggestion_user_not_null.py
new file mode 100644
--- /dev/null
+++ b/pootle/apps/pootle_store/migrations/0002_make_suggestion_user_not_null.py
@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+from django.conf import settings
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pootle_store', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='suggestion',
+            name='user',
+            field=models.ForeignKey(related_name='suggestions', to=settings.AUTH_USER_MODEL),
+            preserve_default=True,
+        ),
+    ]
diff --git a/pootle/apps/pootle_store/models.py b/pootle/apps/pootle_store/models.py
--- a/pootle/apps/pootle_store/models.py
+++ b/pootle/apps/pootle_store/models.py
@@ -135,7 +135,7 @@ class Suggestion(models.Model, base.TranslationUnit):
     target_f = MultiStringField()
     target_hash = models.CharField(max_length=32, db_index=True)
     unit = models.ForeignKey('pootle_store.Unit')
-    user = models.ForeignKey(settings.AUTH_USER_MODEL, null=True,
+    user = models.ForeignKey(settings.AUTH_USER_MODEL, null=False,
                              related_name='suggestions', db_index=True)
     reviewer = models.ForeignKey(settings.AUTH_USER_MODEL, null=True,
                                  related_name='reviews', db_index=True)


Test patch:
Repository: translate/pootle

Issue:
Make redis version check CRITICAL
If the redis version is not sufficiently recent the site will fail.

Currently there is a CRITICAL check for the presence of redis, but the version check is only a WARNING - i think it should be CRITICAL

The hint should also say what the minimum redis version is



Discussion:

Resolving patch:
diff --git a/pootle/checks.py b/pootle/checks.py
--- a/pootle/checks.py
+++ b/pootle/checks.py
@@ -94,6 +94,17 @@ def check_redis(app_configs=None, **kwargs):
             id="pootle.C001",
         ))
     else:
+        redis_version = tuple(int(x) for x
+                              in (queue.connection
+                                       .info()["redis_version"].split(".")))
+        if redis_version < REDIS_MINIMUM_REQUIRED_VERSION:
+            errors.append(checks.Critical(
+                _("Your version of Redis is too old."),
+                hint=_("Update your system's Redis server package to at least "
+                       "version %s" % str(REDIS_MINIMUM_REQUIRED_VERSION)),
+                id="pootle.C007",
+            ))
+
         if len(queue.connection.smembers(Worker.redis_workers_keys)) == 0:
             # We need to check we're not running manage.py rqworker right now..
             import sys
@@ -104,14 +115,6 @@ def check_redis(app_configs=None, **kwargs):
                     id="pootle.W001",
                 ))
 
-        redis_version = queue.connection.info()["redis_version"].split(".")
-        if tuple(int(x) for x in redis_version) < REDIS_MINIMUM_REQUIRED_VERSION:
-            errors.append(checks.Warning(
-                _("Your version of Redis is too old."),
-                hint=_("Update your system's Redis server package"),
-                id="pootle.W002",
-            ))
-
     return errors
 
 


Test patch:
Repository: translate/pootle

Issue:
False positive in `check_duplicate_emails`
The newly-introduced `check_duplicate_emails` check goes through all users without excluding users that are fine to have _empty_ or invalid email addresses (_nobody_ and _default_), hence resulting in a false positive test.



Discussion:
hi @julen 

The problem was that we would like to make User.email unique (see #4035) - and im not entirely sure how to handle these system users.

Perhaps for now we should just exclude them from this check.

That's what I suggest, either use `hide_permission_users()`, or `hide_meta()` if the _system_ user can be taken out as well.

IMHO _system_ user should be left out as well.


Resolving patch:
diff --git a/pootle/apps/accounts/management/commands/find_duplicate_emails.py b/pootle/apps/accounts/management/commands/find_duplicate_emails.py
--- a/pootle/apps/accounts/management/commands/find_duplicate_emails.py
+++ b/pootle/apps/accounts/management/commands/find_duplicate_emails.py
@@ -25,7 +25,8 @@ def handle(self, *args, **kwargs):
             return
 
         for email in duplicates:
-            users = User.objects.filter(email=email).order_by("-last_login")
+            users = (User.objects.hide_meta()
+                                 .filter(email=email).order_by("-last_login"))
             if email:
                 self.stdout.write("The following users have the email: %s\n"
                                   % email)
diff --git a/pootle/apps/accounts/utils.py b/pootle/apps/accounts/utils.py
--- a/pootle/apps/accounts/utils.py
+++ b/pootle/apps/accounts/utils.py
@@ -390,10 +390,11 @@ def verify_user(user):
 def get_duplicate_emails():
     """Get a list of emails that occur more than once in user accounts.
     """
-    return (get_user_model().objects.values('email')
-                            .annotate(Count('email'))
-                            .filter(email__count__gt=1)
-                            .values_list("email", flat=True))
+    return (get_user_model().objects.hide_meta()
+                                    .values('email')
+                                    .annotate(Count('email'))
+                                    .filter(email__count__gt=1)
+                                    .values_list("email", flat=True))
 
 
 def validate_email_unique(email, for_user=None):


Test patch:
Repository: translate/pootle

Issue:
Elasticsearch 1.7.0 breaks Pootle if ES not set up
With `elasticsearch==1.7.0` doing `es.ping()` on a not-setup server raises `ConnectionError`.

This breaks the default install here:

https://github.com/translate/pootle/blob/master/pootle/core/search/backends/elasticsearch.py#L29

Elasticsearch 1.7.0 breaks Pootle if ES not set up
With `elasticsearch==1.7.0` doing `es.ping()` on a not-setup server raises `ConnectionError`.

This breaks the default install here:

https://github.com/translate/pootle/blob/master/pootle/core/search/backends/elasticsearch.py#L29



Discussion:

Resolving patch:
diff --git a/pootle/core/search/backends/elasticsearch.py b/pootle/core/search/backends/elasticsearch.py
--- a/pootle/core/search/backends/elasticsearch.py
+++ b/pootle/core/search/backends/elasticsearch.py
@@ -13,6 +13,7 @@
 
 try:
     from elasticsearch import Elasticsearch
+    from elasticsearch.exceptions import ConnectionError
 except:
     Elasticsearch = None
 
@@ -26,7 +27,12 @@ def __init__(self, config_name):
         self._create_index_if_missing()
 
     def _server_setup_and_alive(self):
-        return self._es is not None and self._es.ping()
+        if self._es is None:
+            return False
+        try:
+            return self._es.ping()
+        except ConnectionError:
+            return False
 
     def _get_es_server(self):
         if self._settings is None or Elasticsearch is None:


Test patch:
Repository: translate/pootle

Issue:
Add syntax highlighting editor to our static pages
Our static pages are HTML, and it would be nice to have HTML syntax highlight in there.

My personal strong preference is http://codemirror.net/ since it is very solid, has a great API, and allows to easily write custom highlighters, which we can make the use of in other places in the app.



Discussion:
Note the end result is always HTML, however the source text can either be HTML, Markdown or reStructuredText, as configured by server admins.

1. Technically, Codemirror has support for both [Markdown](http://codemirror.net/mode/markdown/) and [reStructuredText](http://codemirror.net/mode/rst/) in case someone wants to support that.
2. I explained in #3766 why having anything but HTML is a bad idea. Having to support multiple syntax highlighters is one of the reasons against non-HTML formatters.

I'm fully aware CodeMirror supports the afore-mentioned formats as well as many more — I was commenting on the topic for the sake of completeness, as the original message was leaving behind the fact that other markup formats are currently supported by Pootle.


Resolving patch:
diff --git a/pootle/apps/pootle_misc/context_processors.py b/pootle/apps/pootle_misc/context_processors.py
--- a/pootle/apps/pootle_misc/context_processors.py
+++ b/pootle/apps/pootle_misc/context_processors.py
@@ -10,6 +10,7 @@
 from django.conf import settings
 from django.utils import translation
 
+from pootle.core.markup import get_markup_filter_name
 from pootle.core.utils.json import jsonify
 from pootle_language.models import Language
 from pootle_project.models import Project
@@ -47,6 +48,7 @@ def pootle_context(request):
             'POOTLE_INSTANCE_ID': settings.POOTLE_INSTANCE_ID,
             'POOTLE_CONTACT_ENABLED': (settings.POOTLE_CONTACT_ENABLED and
                                        settings.POOTLE_CONTACT_EMAIL),
+            'POOTLE_MARKUP_FILTER': get_markup_filter_name(),
             'POOTLE_SIGNUP_ENABLED': settings.POOTLE_SIGNUP_ENABLED,
             'SCRIPT_NAME': settings.SCRIPT_NAME,
             'POOTLE_CACHE_TIMEOUT': settings.POOTLE_CACHE_TIMEOUT,
diff --git a/pootle/apps/staticpages/views.py b/pootle/apps/staticpages/views.py
--- a/pootle/apps/staticpages/views.py
+++ b/pootle/apps/staticpages/views.py
@@ -65,6 +65,8 @@ def get_form(self, form_class):
         if self.page_type == ANN_TYPE:
             form.fields['virtual_path'].help_text = u'/pages/' + ANN_VPATH
 
+        form.fields['body'].widget.attrs['class'] = 'js-staticpage-editor'
+
         return form
 
     def form_valid(self, form):


Test patch: