Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Email Rule ignores subfolders
### Description

Mail Rules are all working fine - except if I use a subfolder.
i.e. "INBOX.Archiv" for one of my Email accounts end in
`[2022-04-18 16:08:51,911] [ERROR] [paperless_mail] Rule CMRU.cmru paperless ordner: Error while processing rule: Rule CMRU.cmru paperless ordner: Folder INBOX.Archiv does not exist in account CMRU`

Of cource I tried different spellings and different folders and different Email-Accounts. For all account only the main folder is working

### Expected behavior

Mail rule should process only email in defined subfolder

### Steps to reproduce

see above

### Webserver logs

```bash
[2022-04-18 16:08:51,862] [DEBUG] [paperless_mail] Account CMRU: Processing 1 rule(s)

[2022-04-18 16:08:51,867] [DEBUG] [paperless_mail] Rule CMRU.cmru paperless ordner: Selecting folder INBOX.Archiv

[2022-04-18 16:08:51,911] [ERROR] [paperless_mail] Rule CMRU.cmru paperless ordner: Error while processing rule: Rule CMRU.cmru paperless ordner: Folder INBOX.Archiv does not exist in account CMRU

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 192, in handle_mail_rule

    M.folder.set(rule.folder)

  File "/usr/local/lib/python3.9/site-packages/imap_tools/folder.py", line 44, in set

    check_command_status(result, MailboxFolderSelectError)

  File "/usr/local/lib/python3.9/site-packages/imap_tools/utils.py", line 45, in check_command_status

    raise exception(command_result=command_result, expected=expected)

imap_tools.errors.MailboxFolderSelectError: Response status "OK" expected, but "NO" received. Data: [b"INBOX.Archiv doesn't exist."]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 177, in handle_mail_account

    total_processed_files += self.handle_mail_rule(M, rule)

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 194, in handle_mail_rule

    raise MailError(

paperless_mail.mail.MailError: Rule CMRU.cmru paperless ordner: Folder INBOX.Archiv does not exist in account CMRU
```


### Screenshots

_No response_

### Paperless-ngx version

1.6.0

### Host OS

Synology/ Docker

### Installation method

Docker

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Did you try different separators? Eg #550 
"I tried ". , | / \" 
Just to be sure, the expected folder is named `INBOX.Archiv` not `INBOX.Archive` (the extra e there)?
> Just to be sure, the expected folder is named `INBOX.Archiv` not `INBOX.Archive` (the extra e there)?

Good question, but yes, it`s German :-) - so "ARCHIV" is correct.

Meanwhile I found the following:

- If I choose only "Archiv" (wihout INBOX) it works. 
- This works only for folders on "top level", 
- this might be specific for MS exchange, because "posteingang" (German for INBOX) seems is NOT top level of  INBOX =:-\ - my "Archiv" is in the list of folders at the same level as "INBOX/Posteingang"
- but for each subfolder, no seperator seems to work, neither the subfolder is in "Archiv" nor in "Posteingang"

Despite IMAP being a standard, things really aren't standardized so much.  We can't really account for every variance in how mail servers decide folder structure is, but I think we could try to surface more information to the user in this error case.

I'm thinking in this case, where folder selection fails, we try to list out the account folders visible, before raising up the error.  Then a user can hopefully locate the folder they wanted in the listing.

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -190,6 +190,21 @@ def handle_mail_rule(self, M, rule):
         try:
             M.folder.set(rule.folder)
         except MailboxFolderSelectError:
+
+            self.log(
+                "error",
+                f"Unable to access folder {rule.folder}, attempting folder listing",
+            )
+            try:
+                for folder_info in M.folder.list():
+                    self.log("info", f"Located folder: {folder_info.name}")
+            except Exception as e:
+                self.log(
+                    "error",
+                    "Exception during folder listing, unable to provide list folders: "
+                    + str(e),
+                )
+
             raise MailError(
                 f"Rule {rule}: Folder {rule.folder} "
                 f"does not exist in account {rule.account}",


Test patch:
diff --git a/src/paperless_mail/tests/test_mail.py b/src/paperless_mail/tests/test_mail.py
--- a/src/paperless_mail/tests/test_mail.py
+++ b/src/paperless_mail/tests/test_mail.py
@@ -15,6 +15,7 @@
 from documents.models import Correspondent
 from documents.tests.utils import DirectoriesMixin
 from imap_tools import EmailAddress
+from imap_tools import FolderInfo
 from imap_tools import MailboxFolderSelectError
 from imap_tools import MailMessage
 from imap_tools import MailMessageFlags
@@ -53,13 +54,12 @@ def __exit__(self, exc_type, exc_val, exc_tb):
     def __init__(self):
         self.messages: List[MailMessage] = []
         self.messages_spam: List[MailMessage] = []
+        self.folder = BogusFolderManager()
 
     def login(self, username, password):
         if not (username == "admin" and password == "secret"):
             raise Exception()
 
-    folder = BogusFolderManager()
-
     def fetch(self, criteria, mark_seen, charset=""):
         msg = self.messages
 
@@ -621,6 +621,72 @@ def test_error_skip_rule(self):
         self.assertEqual(len(self.bogus_mailbox.messages), 2)
         self.assertEqual(len(self.bogus_mailbox.messages_spam), 1)
 
+    def test_error_folder_set(self):
+        """
+        GIVEN:
+            - Mail rule with non-existent folder
+        THEN:
+            - Should call list to output all folders in the account
+            - Should not process any messages
+        """
+        account = MailAccount.objects.create(
+            name="test2",
+            imap_server="",
+            username="admin",
+            password="secret",
+        )
+        _ = MailRule.objects.create(
+            name="testrule",
+            account=account,
+            action=MailRule.AttachmentAction.MOVE,
+            action_parameter="spam",
+            filter_subject="Claim",
+            order=1,
+            folder="uuuhhhh",  # Invalid folder name
+        )
+
+        self.bogus_mailbox.folder.list = mock.Mock(
+            return_value=[FolderInfo("SomeFoldername", "|", ())],
+        )
+
+        self.mail_account_handler.handle_mail_account(account)
+
+        self.bogus_mailbox.folder.list.assert_called_once()
+        self.assertEqual(self.async_task.call_count, 0)
+
+    def test_error_folder_set_error_listing(self):
+        """
+        GIVEN:
+            - Mail rule with non-existent folder
+            - Mail account folder listing raises exception
+        THEN:
+            - Should not process any messages
+        """
+        account = MailAccount.objects.create(
+            name="test2",
+            imap_server="",
+            username="admin",
+            password="secret",
+        )
+        _ = MailRule.objects.create(
+            name="testrule",
+            account=account,
+            action=MailRule.AttachmentAction.MOVE,
+            action_parameter="spam",
+            filter_subject="Claim",
+            order=1,
+            folder="uuuhhhh",  # Invalid folder name
+        )
+
+        self.bogus_mailbox.folder.list = mock.Mock(
+            side_effect=MailboxFolderSelectError(None, "uhm"),
+        )
+
+        self.mail_account_handler.handle_mail_account(account)
+
+        self.bogus_mailbox.folder.list.assert_called_once()
+        self.assertEqual(self.async_task.call_count, 0)
+
     @mock.patch("paperless_mail.mail.MailAccountHandler.get_correspondent")
     def test_error_skip_mail(self, m):
         def get_correspondent_fake(message, rule):