Repository: Mailu/Mailu

Issue:
Doveadm Cannot List Users
<!--

Thank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.
For **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).

To be able to help you best, we need some more information.

Before you open your issue
- Check if no issue or pull-request for this already exists.
- Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)
- You understand `Mailu` is made by volunteers in their **free time** — be concise, civil and accept that delays can occur.
- The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.

Please put your text outside of the comment blocks to be visible. You can use the button "Preview" above to check.

-->

## Environment & Version

### Environment

- [x] docker-compose
- [ ] kubernetes
- [ ] docker swarm

### Version

- Version: `1.9`

## Description
`doveadm search -A` doesn't work

## Replication Steps
Deploy a fresh new server, add a few users and execute `doveadm search -A ALL` in `imap` container.

## Observed behaviour
Nothing showed.

## Expected behaviour
Should give a list of existing mails.

## Extra Info
If I run `doveadm -Dv search -u user@domain.net ALL`, it succeeded.

## Logs
Result of `doveadm -Dv search -A ALL`
```
Oct 31 15:58:26 Debug: Loading modules from directory: /usr/lib/dovecot
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/lib10_quota_plugin.so
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/lib20_fts_plugin.so
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/lib20_quota_clone_plugin.so
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/lib21_fts_xapian_plugin.so
Oct 31 15:58:26 Debug: Loading modules from directory: /usr/lib/dovecot/doveadm
Oct 31 15:58:26 Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: acl_rights_update_import: symbol not found (this is usually intentional, so just ignore this message)
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Oct 31 15:58:26 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so
Oct 31 15:58:26 Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: mail_crypt_box_get_public_key: symbol not found (this is usually intentional, so just ignore this message)
Oct 31 15:58:26 Debug: auth-master: userdb list: Started listing users (user_mask=)
Oct 31 15:58:26 Debug: auth-master: conn unix:/run/dovecot/auth-userdb: Connecting
Oct 31 15:58:26 Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=1,uid=0): Client connected (fd=9)
Oct 31 15:58:26 Debug: auth-master: userdb list: Finished listing users
Oct 31 15:58:26 Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=1,uid=0): Disconnected: Connection closed (fd=9)
```

Result of `doveadm -Dv search -u user@domain.net ALL`
```
Oct 31 16:04:46 Debug: Loading modules from directory: /usr/lib/dovecot
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/lib10_quota_plugin.so
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/lib20_fts_plugin.so
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/lib20_quota_clone_plugin.so
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/lib21_fts_xapian_plugin.so
Oct 31 16:04:46 Debug: Loading modules from directory: /usr/lib/dovecot/doveadm
Oct 31 16:04:46 Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: acl_rights_update_import: symbol not found (this is usually intentional, so just ignore this message)
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Oct 31 16:04:46 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so
Oct 31 16:04:46 Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: mail_crypt_box_get_public_key: symbol not found (this is usually intentional, so just ignore this message)
Oct 31 16:04:46 doveadm(user@domain.net)<175><>: Debug: auth-master: userdb lookup(user@domain.net): Started userdb lookup
Oct 31 16:04:46 doveadm(user@domain.net)<175><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb: Connecting
Oct 31 16:04:46 doveadm(user@domain.net)<175><>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=1,uid=0): Client connected (fd=9)
Oct 31 16:04:46 doveadm(user@domain.net)<175><>: Debug: auth-master: userdb lookup(user@domain.net): auth USER input: user@domain.net quota_rule=*:bytes=1000000000
Oct 31 16:04:46 doveadm(user@domain.net)<175><>: Debug: auth-master: userdb lookup(user@domain.net): Finished userdb lookup (username=user@domain.net quota_rule=*:bytes=1000000000)
Oct 31 16:04:46 doveadm(user@domain.net)<175><>: Debug: Added userdb setting: plugin/quota_rule=*:bytes=1000000000
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: Effective uid=8, gid=12, home=/mail/user@domain.net
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: Quota root: name=User quota backend=count args=
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: Quota rule: root=User quota mailbox=* bytes=1000000000 messages=0
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: Quota grace: root=User quota bytes=100000000 (10%)
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/mail/user@domain.net
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: maildir++: root=/mail/user@domain.net, index=, indexpvt=, control=, inbox=/mail/user@domain.net, alt=
Oct 31 16:04:46 doveadm(user@domain.net)<175><nWh4MR7yX2OvAAAAU8qUbw>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping
Oct 31 16:04:46 doveadm(user@domain.net): Debug: Mailbox Trash: Mailbox opened because: search
Oct 31 16:04:46 doveadm(user@domain.net): Debug: Mailbox Junk: Mailbox opened because: search
Oct 31 16:04:46 doveadm(user@domain.net): Debug: Mailbox Sent: Mailbox opened because: search
Oct 31 16:04:46 doveadm(user@domain.net): Debug: Mailbox Drafts: Mailbox opened because: search
Oct 31 16:04:46 doveadm(user@domain.net): Debug: Mailbox INBOX: Mailbox opened because: search
Oct 31 16:04:46 doveadm(user@domain.net): Debug: dict(proxy)<user@domain.net>: Waiting for dict to finish pending operations
Oct 31 16:04:46 doveadm(user@domain.net): Debug: dict(proxy)<user@domain.net>: Waiting for dict to finish pending operations
8e5bb0397c244b630f351500ed1cffc7 1
8e5bb0397c244b630f351500ed1cffc7 2
8e5bb0397c244b630f351500ed1cffc7 3
8e5bb0397c244b630f351500ed1cffc7 4
8e5bb0397c244b630f351500ed1cffc7 5
...
Oct 31 16:04:46 doveadm(175): Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=1,uid=0): Disconnected: Connection closed (fd=9)
```



Discussion:

Resolving patch:
diff --git a/core/admin/mailu/internal/views/dovecot.py b/core/admin/mailu/internal/views/dovecot.py
--- a/core/admin/mailu/internal/views/dovecot.py
+++ b/core/admin/mailu/internal/views/dovecot.py
@@ -5,6 +5,7 @@
 import flask
 import socket
 import os
+import sqlalchemy.exc
 
 @internal.route("/dovecot/passdb/<path:user_email>")
 def dovecot_passdb_dict(user_email):
@@ -19,12 +20,20 @@ def dovecot_passdb_dict(user_email):
         "allow_nets": ",".join(allow_nets)
     })
 
+@internal.route("/dovecot/userdb/")
+def dovecot_userdb_dict_list():
+    return flask.jsonify([
+        user[0] for user in models.User.query.filter(models.User.enabled.is_(True)).with_entities(models.User.email).all()
+    ])
 
 @internal.route("/dovecot/userdb/<path:user_email>")
 def dovecot_userdb_dict(user_email):
-    user = models.User.query.get(user_email) or flask.abort(404)
+    try:
+        quota = models.User.query.filter(models.User.email==user_email).with_entities(models.User.quota_bytes).one_or_none() or flask.abort(404)
+    except sqlalchemy.exc.StatementError as exc:
+        flask.abort(404)
     return flask.jsonify({
-        "quota_rule": "*:bytes={}".format(user.quota_bytes)
+        "quota_rule": f"*:bytes={quota[0]}"
     })
 
 
diff --git a/core/base/libs/podop/podop/dovecot.py b/core/base/libs/podop/podop/dovecot.py
--- a/core/base/libs/podop/podop/dovecot.py
+++ b/core/base/libs/podop/podop/dovecot.py
@@ -40,6 +40,7 @@ def __init__(self, table_map):
     def connection_made(self, transport):
         logging.info('Connect {}'.format(transport.get_extra_info('peername')))
         self.transport = transport
+        self.transport_lock = asyncio.Lock()
 
     def data_received(self, data):
         logging.debug("Received {}".format(data))
@@ -77,10 +78,11 @@ def process_hello(self, major, minor, value_type, user, dict_name):
         logging.debug("Client {}.{} type {}, user {}, dict {}".format(
             self.major, self.minor, self.value_type, self.user, dict_name))
 
-    async def process_lookup(self, key, user=None):
+    async def process_lookup(self, key, user=None, is_iter=False):
         """ Process a dict lookup message
         """
         logging.debug("Looking up {} for {}".format(key, user))
+        orig_key = key
         # Priv and shared keys are handled slighlty differently
         key_type, key = key.decode("utf8").split("/", 1)
         try:
@@ -93,9 +95,38 @@ async def process_lookup(self, key, user=None):
                 response = result
             else:
                 response = json.dumps(result).encode("ascii")
-            return self.reply(b"O", response)
+            return await (self.reply(b"O", orig_key, response) if is_iter else self.reply(b"O", response))
         except KeyError:
-            return self.reply(b"N")
+            return await self.reply(b"N")
+
+    async def process_iterate(self, flags, max_rows, path, user=None):
+        """ Process an iterate command
+        """
+        logging.debug("Iterate flags {} max_rows {} on {} for {}".format(flags, max_rows, path, user))
+        # Priv and shared keys are handled slighlty differently
+        key_type, key = path.decode("utf8").split("/", 1)
+        max_rows = int(max_rows.decode("utf-8"))
+        flags = int(flags.decode("utf-8"))
+        if flags != 0: # not implemented
+            return await self.reply(b"F")
+        rows = []
+        try:
+            result = await self.dict.iter(key)
+            logging.debug("Found {} entries: {}".format(len(result), result))
+            for i,k in enumerate(result):
+                if max_rows > 0 and i >= max_rows:
+                    break
+                rows.append(self.process_lookup((path.decode("utf8")+k).encode("utf8"), user, is_iter=True))
+            await asyncio.gather(*rows)
+            async with self.transport_lock:
+                self.transport.write(b"\n") # ITER_FINISHED
+            return
+        except KeyError:
+            return await self.reply(b"F")
+        except Exception as e:
+            for task in rows:
+                task.cancel()
+            raise e
 
     def process_begin(self, transaction_id, user=None):
         """ Process a dict begin message
@@ -124,13 +155,14 @@ async def process_commit(self, transaction_id):
         # Remove stored transaction
         del self.transactions[transaction_id]
         del self.transactions_user[transaction_id]
-        return self.reply(b"O", transaction_id)
+        return await self.reply(b"O", transaction_id)
 
-    def reply(self, command, *args):
-        logging.debug("Replying {} with {}".format(command, args))
-        self.transport.write(command)
-        self.transport.write(b"\t".join(map(tabescape, args)))
-        self.transport.write(b"\n")
+    async def reply(self, command, *args):
+        async with self.transport_lock:
+            logging.debug("Replying {} with {}".format(command, args))
+            self.transport.write(command)
+            self.transport.write(b"\t".join(map(tabescape, args)))
+            self.transport.write(b"\n")
 
     @classmethod
     def factory(cls, table_map):
@@ -141,6 +173,7 @@ def factory(cls, table_map):
     COMMANDS = {
         ord("H"): process_hello,
         ord("L"): process_lookup,
+        ord("I"): process_iterate,
         ord("B"): process_begin,
         ord("C"): process_commit,
         ord("S"): process_set


Test patch: