Repository: WeblateOrg/weblate

Issue:
Repetition check
Some platforms/tools have checks for repetition, which is the error that happens most often.

When something is very complex, it is easy to stop, think, and then carry on from the word already put down onwards. It can also happen when rewriting a sentence, ending up with two parts that share a repeated word in its midst.

For example this this is an example of a repeated word.

Example and example, fine. "This this" etc. is almost always an error.
If not an outright error, it is often best avoided, so quality would improve a lot overall with a check that (in most languages) has very few false positives.

<bountysource-plugin>

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/90869677-repetition-check?utm_campaign=plugin&utm_content=tracker%2F253393&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F253393&utm_medium=issues&utm_source=github).
</bountysource-plugin>


Discussion:
Should be easy to implement, for example using https://www.oreilly.com/library/view/regular-expressions-cookbook/9780596802837/ch05s08.html
Hi, I'd be interested in taking this on. Where should this check live? What are the input and output? For example, should this check simply report that there is a (probably unintentional) repetition, or should it attempt to rectify the repetition?
It should live in weblate.checks, see 8d4764cb74df405461d7a20a159c050e1de18b3c for example of adding a new check code.
I'm implementing the regex check, but I'd like to raise the relevant linguistic issues below so I understand the use case better.

~

Reduplication is common in a fair number of languages. For example, in Persian, one way to say "very big" is by saying "big" twice [1]:

بزرگ بزرگ

So we might think that we can enable this check only for content words, or only in certain languages. But in English, I can say "very very" (a reduplicated function word) as an intensifier. So, while "this this" is almost certainly an error, it's difficult to develop a heuristic (e.g. "no reduplicated function words") that applies across the board.

[1] There is actually a pronounced morpheme between these two (it's "bozorg-e bozorg"), but, orthographically, it's just a repeated token. 

Resolving patch:
diff --git a/weblate/checks/duplicate.py b/weblate/checks/duplicate.py
new file mode 100644
--- /dev/null
+++ b/weblate/checks/duplicate.py
@@ -0,0 +1,39 @@
+#
+# Copyright © 2012 - 2020 Michal Čihař <michal@cihar.com>
+#
+# This file is part of Weblate <https://weblate.org/>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+
+
+import re
+
+from django.utils.translation import gettext_lazy as _
+
+from weblate.checks.base import TargetCheck
+
+
+class DuplicateCheck(TargetCheck):
+    """Check for duplicated tokens."""
+
+    check_id = "duplicate"
+    name = _("Consecutive duplicated tokens")
+    description = _("Text contains the same token twice in a row")
+    severity = "warning"
+
+    def check_single(self, source, target, unit):
+        if re.search(r"\b(\w+)(?:\s+\1)+\b", target):
+            return True
+        return False


Test patch:
diff --git a/weblate/checks/tests/test_duplicate_checks.py b/weblate/checks/tests/test_duplicate_checks.py
new file mode 100644
--- /dev/null
+++ b/weblate/checks/tests/test_duplicate_checks.py
@@ -0,0 +1,51 @@
+#
+# Copyright © 2012 - 2020 Michal Čihař <michal@cihar.com>
+#
+# This file is part of Weblate <https://weblate.org/>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+
+"""Tests for duplicate checks."""
+
+
+from weblate.checks.duplicate import DuplicateCheck
+from weblate.checks.tests.test_checks import CheckTestCase, MockUnit
+
+
+class DuplicateCheckTest(CheckTestCase):
+
+    check = DuplicateCheck()
+
+    _MOCK_UNIT = MockUnit(code="cs", note="")
+
+    def _run_check(self, target):
+        return self.check.check_single("", target, self._MOCK_UNIT)
+
+    def test_no_duplicated_token(self):
+        self.assertFalse(self._run_check("I have two lemons"))
+
+    def test_check_respects_boundaries_suffix(self):
+        """'lemon lemon' is a false duplicate."""
+        self.assertFalse(self._run_check("I have two lemon lemons"))
+
+    def test_check_respects_boundaries_prefix(self):
+        """'melon on' is a false duplicate."""
+        self.assertFalse(self._run_check("I have a melon on my back"))
+
+    def test_check_single_duplicated_token(self):
+        self.assertTrue(self._run_check("I have two two lemons"))
+
+    def test_check_multiple_duplicated_tokens(self):
+        self.assertTrue(self._run_check("I have two two lemons lemons"))