Repository: commons-app/apps-android-commons (language: kotlin)

Issue:
[Bug]: Category "Amavenita (ship, 2014)" not found
### Summary

I serached for a category that exists, but it doesn't show up.

### Steps to reproduce

1. When uploading an image, search for category: Amavenita


### Expected behaviour

[Category:Amavenita (ship, 2014)](https://commons.wikimedia.org/wiki/Category:Amavenita_(ship,_2014)) should be suggested.

### Actual behaviour

"No Categories found"

### Device name

Samsung Galaxy S10

### Android version

Android 12

### Commons app version

3.1.1~1c9267ca0

### Device logs

_No response_

### Screen-shots

![Screenshot_20220318-180530_Commons](https://user-images.githubusercontent.com/5566159/159050916-23c318f1-f0bb-4c53-b8b9-01a1b24a1e4f.jpg)


### Would you like to work on the issue?

_No response_


Discussion hints:
@Ainali isn't this the expected behaviour? Because, Amavenita (ship, 2014) is a non-hidden category and I think we are only supposed to show the hidden category in search bar. But I could be wrong. 
Oh, I didn't realize. Then this is a duplicate with #4192 (and a use case for why it is needed, this is how all ships are categorized and the way for a human to identify them). 
> "we are only supposed to show the hidden category in search bar"

@devarsh-mavani-19 I would consider that a bug. If a category is not hidden, then the upload wizard's category search step is **supposed to** show it.
By the way, when I search for "rabbits", the first result is https://commons.wikimedia.org/wiki/Category:Rabbits which is not a hidden category:

![adb](https://user-images.githubusercontent.com/99590/159115296-dff3836c-346d-4088-b66f-00e2c4f8d59e.png)

> By the way, when I search for "rabbits", the first result is https://commons.wikimedia.org/wiki/Category:Rabbits which is not a hidden category:
> 
> ![adb](https://user-images.githubusercontent.com/99590/159115296-dff3836c-346d-4088-b66f-00e2c4f8d59e.png)

Yes that is how it should work if it is non-hidden, then it should be displayed. However, in case of Ainali it was not displayed because category name contained a year which was older than previous year  (Amavenita (ship, **2014**)). so according to this issue #750 it should be filtered out.

so just to summarize currently 2 conditions must met 1. it should be non-hidden and 2nd either should not have any year in name or the year should be current or previous year.
> However, in case of Ainali it was not displayed because category name contained a year which was older than previous year (Amavenita (ship, **2014**)). so according to this issue #750 it should be filtered out.

Then I would still consider that issue to have been too poorly defined, because the app shouldn't filter out things where the year is part of the identifier of that thing, in this case, the ship that got the name in 2014. (Also, the app should probably not filter out by only older than current year, since it's common to upload files a day or two after they are taken which means that if I upload an image on January 1, I expect to see the category for the year before. Safer would be to filter categories where the year is less than current year minus 2.)
I'm presuming the issue #750 was understood properly but the fix for it was a bit loose than it should've been. The [relevant code snippet](https://github.com/commons-app/apps-android-commons/blob/15fa93e8d004bd0f55515e6d33c7228b23d63026/app/src/main/java/fr/free/nrw/commons/category/CategoriesModel.kt):

```kotlin
return item.matches(".*(19|20)\\d{2}.*".toRegex())
                && !item.contains(yearInString)
                && !item.contains(prevYearInString)
                || item.matches("(.*)needing(.*)".toRegex())
                || item.matches("(.*)taken on(.*)".toRegex())
                || item.matches(".*0s.*".toRegex())
                && !item.matches(".*(200|201)0s.*".toRegex())
```

That essentially returns true (which means the category is filtered out) when the name:

1. Contains "19xx" OR "20xx" AND
2. Does not contain current year AND
3. Does not contain previous year OR
4. Contains "needing" OR
5. Contains "take on" OR
6. Contains 0s AND
7. Does not contain "2000s" OR "2010s"

If we were to convert the implicit precedence to explicit ones the condition looks like this:

(((1) AND (2) AND (3)) OR (4) OR (5) OR ((6) AND (7)))

Given [short-circuiting evaluation](https://en.wikipedia.org/wiki/Short-circuit_evaluation), the condition should've returned `true` for "Amavenita (ship, 2014)" thus the category getting filtered out.

We could fix this by fixing the loose condition. I'll open a PR.
Thanks a lot @devarsh-mavani-19 for finding this out!
I had forgotten about this feature.
Do we all agree on this?
- `Photographs taken on 2015-11-08` -> Hide
- `Amavenita (ship, 2014)` -> Do not hide
> Do we all agree on this?

agree
What is the status of this?
@Discostu36 This bug still happens, I just tried on the latest Git code. Do you want to work on it? :-)
Sadly, I lack the skills. I have started learning Kotlin development, but I'm only at the very first baby steps (and don't have much time).
For those interested in picking this up, MR #4902 already has the changes to fix this. Only the tests need fixing IIRC. So, it could be used as a base.

I intend to get around to completing it. Do feel free to beat me to it, though 😉 
I want to try ✋
Apart from filtering strings `needing` & `taken on` we have these filters 

- years that ends with XXX0s & are above 2000s are allowed i.e 2000, 2010, 2020.
- Otherwise we only allow `current` or `previous year`.

Now, this issue needs the second condition to be removed i.e any year should be allowed. But then it also doesn't make sense to have filter for decades (in the first point) right? 

### Solution: 

We can remove both of the filters and allow any year between 19XX & 20XX (inclusive) 

OR

Just have one filter to allow any year above >2000s instead of only `current` or `previous` year or in `decades` terms.

WDYT? @nicolas-raoul 

Also, api only works correctly under `prod` because the query (Amavenita) mentioned in the issue was returning empty result in `beta` , shouldn't be an issue right?


@parneet-guraya If I understand correctly Sivaraam's comment above, only the unit tests need fixing. :-)

Further modifications might be worth a different issue.

Thanks!
> So, I'm starting to do you think if we could just take forward a combination of your changes (https://github.com/commons-app/apps-android-commons/pull/5761) and the one proposed in https://github.com/commons-app/apps-android-commons/pull/4902 in the next release and see how it goes 🤔

As per [comment](https://github.com/commons-app/apps-android-commons/issues/5749#issuecomment-2227279839), solution for this #4902 overlaps with the PR that was later merged #5761. I had to solve the conflict but solution made the other filters redundant that's why the comment (my comment above). 

>Now, this issue needs the second condition to be removed i.e any year should be allowed. But then it also doesn't make sense to have filter for decades (in the first point) right?
I get what you're trying to say @parneet-guraya. The fix done via #5761 does conflict with the PR #4902 significantly. I think the change does need a bit of tweak as a consequence. I've merged recent main and tweaked the fix accordingly. Hope this helps you show a way out of the conflict.

Pull request: Enhance filtering criteria for category search
Fixes #4901

What changes did you make and why?

This PR is an extension of #4902 PR. refer to [comment](https://github.com/commons-app/apps-android-commons/issues/4901#issuecomment-2592110652). Conflicting changes were made while this PR was open making changes introduced outdated. Also, some of the checks were redundant so that's why started fresh with this PR.

Now, spam filtering function filter these strings ->

- that contains `needing` or `taken on`
- that contains dates in format `dd mm yyyy or yy`  and `mm/yyyy or yy`  (in any order possible)  with separators `/` `.` `-` in between.
- that contains years that doesn't come in 20XX range.

- [X] Wrote tests.

**Example input and output (true if filtered) ->** 
```

Media_needing_categories_as_of_16_June_2017 -> true
Photo_taken on_23_March_2009 -> true
Photo_taken_12/10/2023 -> true
Event_on_25-12-19 -> true
Image_in_22/04/17 -> true
Video_in_2023-05 -> true
Pic_taken_20-12 -> true
Article_in_2024 -> false
Photo_in_1998 -> true
No_date_category -> false
17/04/2021 -> true
Captured_in_1980 -> true
File_uploaded_in_18-05 -> true
Shot_2019/10/15 -> true
Just_some_random_text -> false
Updated_in_21/07 -> true

```
 



Test command: ./gradlew :app:testBetaDebugUnitTest --tests "fr.free.nrw.commons.category.CategoriesModelTest"

Fix patch:
diff --git a/app/.attach_pid781771 b/app/.attach_pid781771
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/app/src/main/java/fr/free/nrw/commons/category/CategoriesModel.kt b/app/src/main/java/fr/free/nrw/commons/category/CategoriesModel.kt
index fd90be95fd..47147944c6 100644
--- a/app/src/main/java/fr/free/nrw/commons/category/CategoriesModel.kt
+++ b/app/src/main/java/fr/free/nrw/commons/category/CategoriesModel.kt
@@ -36,37 +36,35 @@ class CategoriesModel
          * @return
          */
         fun isSpammyCategory(item: String): Boolean {
-            // Check for current and previous year to exclude these categories from removal
-            val now = Calendar.getInstance()
-            val curYear = now[Calendar.YEAR]
-            val curYearInString = curYear.toString()
-            val prevYear = curYear - 1
-            val prevYearInString = prevYear.toString()
-            Timber.d("Previous year: %s", prevYearInString)
-
-            val mentionsDecade = item.matches(".*0s.*".toRegex())
-            val recentDecade = item.matches(".*20[0-2]0s.*".toRegex())
-            val spammyCategory =
-                item.matches("(.*)needing(.*)".toRegex()) ||
-                    item.matches("(.*)taken on(.*)".toRegex())
 
             // always skip irrelevant categories such as Media_needing_categories_as_of_16_June_2017(Issue #750)
+            val spammyCategory = item.matches("(.*)needing(.*)".toRegex())
+                    || item.matches("(.*)taken on(.*)".toRegex())
+
+            // checks for
+            // dd/mm/yyyy or yy
+            // yyyy or yy/mm/dd
+            // yyyy or yy/mm
+            // mm/yyyy or yy
+            // for `yy` it is assumed that 20XX is implicit.
+            // with separators [., /, -]
+            val isIrrelevantCategory =
+                item.contains("""\d{1,2}[-/.]\d{1,2}[-/.]\d{2,4}|\d{2,4}[-/.]\d{1,2}[-/.]\d{1,2}|\d{2,4}[-/.]\d{1,2}|\d{1,2}[-/.]\d{2,4}""".toRegex())
+
+
             if (spammyCategory) {
                 return true
             }
 
-            if (mentionsDecade) {
-                // Check if the year in the form of XX(X)0s is recent/relevant, i.e. in the 2000s or 2010s/2020s as stated in Issue #1029
-                // Example: "2020s" is OK, but "1920s" is not (and should be skipped)
-                return !recentDecade
-            } else {
-                // If it is not an year in decade form (e.g. 19xxs/20xxs), then check if item contains a 4-digit year
-                // anywhere within the string (.* is wildcard) (Issue #47)
-                // And that item does not equal the current year or previous year
-                return item.matches(".*(19|20)\\d{2}.*".toRegex()) &&
-                    !item.contains(curYearInString) &&
-                    !item.contains(prevYearInString)
+            if(isIrrelevantCategory){
+                return true
             }
+
+            val hasYear = item.matches("(.*\\d{4}.*)".toRegex())
+            val validYearsRange = item.matches(".*(20[0-9]{2}).*".toRegex())
+
+            // finally if there's 4 digits year exists in XXXX it should only be in 20XX range.
+            return  hasYear && !validYearsRange
         }
 
         /**


Test patch:
diff --git a/app/src/test/kotlin/fr/free/nrw/commons/category/CategoriesModelTest.kt b/app/src/test/kotlin/fr/free/nrw/commons/category/CategoriesModelTest.kt
index 8c336470a6..21fdba2f57 100644
--- a/app/src/test/kotlin/fr/free/nrw/commons/category/CategoriesModelTest.kt
+++ b/app/src/test/kotlin/fr/free/nrw/commons/category/CategoriesModelTest.kt
@@ -11,6 +11,7 @@ import fr.free.nrw.commons.upload.GpsCategoryModel
 import io.reactivex.Single
 import io.reactivex.subjects.BehaviorSubject
 import media
+import org.junit.Assert
 import org.junit.Before
 import org.junit.Test
 import org.mockito.ArgumentMatchers
@@ -331,4 +332,42 @@ class CategoriesModelTest {
             media(),
         )
     }
+
+    @Test
+    fun `test valid input with XXXX in it between the expected range 20XX`() {
+        val input = categoriesModel.isSpammyCategory("Amavenita (ship, 2014)")
+        Assert.assertFalse(input)
+    }
+
+    @Test
+    fun `test valid input with XXXXs in it between the expected range 20XXs`() {
+        val input = categoriesModel.isSpammyCategory("Amavenita (ship, 2014s)")
+        Assert.assertFalse(input)
+    }
+
+    @Test
+    fun `test invalid category when have needing in the input`() {
+        val input = categoriesModel.isSpammyCategory("Media needing categories as of 30 March 2017")
+        Assert.assertTrue(input)
+    }
+
+    @Test
+    fun `test invalid category when have taken on in the input`() {
+        val input = categoriesModel.isSpammyCategory("Photographs taken on 2015-12-08")
+        Assert.assertTrue(input)
+    }
+
+    @Test
+    fun `test invalid category when have yy mm or yy mm dd in the input`() {
+        // filtering based on [., /, -]  separators between the dates.
+        val input = categoriesModel.isSpammyCategory("Image class 09.14")
+        Assert.assertTrue(input)
+    }
+
+    @Test
+    fun `test invalid category when have years not in 20XX range`() {
+        val input = categoriesModel.isSpammyCategory("Japan in the 1400s")
+        Assert.assertTrue(input)
+    }
+
 }