Repository: Mailu/Mailu

Issue:
dovecot: support for zstd/lz4 compression is not compiled in
## Environment & Versions
### Environment
 - [x] docker-compose
 - [ ] kubernetes
 - [ ] docker swarm

### Versions
1.9

## Description
When `COMPRESSION` in `mailu.env` is set to `zstd` or `lz4` and a new mail arrives, the imap container logs something like this:
```
Jan 04 23:32:46 lmtp(mymailaddress@mydomain.com)<32616><kRbzEw7L1GFofwAADGH2HQ>: Error: zlib_save: Support not compiled in for handler: zstd
Jan 04 23:32:46 indexer-worker(mymailaddress@mydomain.com)<32619><D5ZaIcXUkKDAqMsD:Z6zPGQ7L1GFrfwAADGH2HQ>: Error: zlib_save: Support not compiled in for handler: zstd
```

```
Jan 04 23:43:03 imap(mymailaddress@mydomain.com)<178><2CwwXMnUwMXAqMsE>: Error: zlib_save: Support not compiled in for handler: lz4
Jan 04 23:43:03 indexer-worker(mymailaddress@mydomain.com)<180><B4qhJXfN1GGvAAAAflj66Q:/8frM3fN1GG0AAAAflj66Q>: Error: zlib_save: Support not compiled in for handler: lz4
```

As far as I can tell the Dovecot package from Alpine is probably not compiled with zstd/lz4 support, but I'm not sure since the build log of the last Alpine package returns 404 :roll_eyes:

This is __not__ breaking anything, mails will be received, but they won't be compressed.

## Replication Steps
1. in `mailu.env` set `COMPRESSION=zstd` or `COMPRESSION=lz4`
2. `docker-compose down && docker-compose up`
3. send yourself a mail
4. watch the logs of the imap container

## Expected behaviour
With `COMPRESSION=zstd`/`COMPRESSION=lz4` set, mails should be compressed with zstd/lz4 and then stored.

## Workaround
Set `COMPRESSION=gz` if you need compressed mail storage.


Discussion:
Does this mean it never worked?
I also see this functionality is not documented in the configuration reference.

Do  you know which packages provide zstd/lz4 support?
If it is not listed in the build file (makedepends?) then we might have to file an issue in the alpine issue tracker:
https://git.alpinelinux.org/aports/tree/main/dovecot/APKBUILD#n22

For now we could document the option COMPRESSION in the configuration reference and in the mailu.env file remove zstd/lz4 in the comment.
> Does this mean it never worked?

Yes it does. :grimacing:

> Do you know which packages provide zstd/lz4 support?
> If it is not listed in the build file (makedepends?) then we might have to file an issue in the alpine issue tracker:
> https://git.alpinelinux.org/aports/tree/main/dovecot/APKBUILD#n22

No package simply provides it.

For dovecot to support zstd/lz4 compression:
1. `zstd-dev`/`lz4-dev` have to be installed _before_ dovecot is compiled (so they would have to be in the `makedepends` list)
2. the configure flags `--with-zstd` and `--with-lz4` should be supplied (it may auto-detect that the needed headers are present)

Both these steps are not done in Alpine's APKBUILD - since these compression features are optional and Alpine enjoys minimalism, I'm not sure if they would count that as a bug/a missing feature/in need of a fix …

> For now we could document the option COMPRESSION in the configuration reference and in the mailu.env file remove zstd/lz4 in the comment.

Yep, the only working COMPRESSION options are `gz`/`bz2` since only `zlib-dev` and `bzip2-dev` are in `makedepends`.
This was by the way also true before https://github.com/Mailu/Mailu/commit/2316ef1162d400fc486ad72df62fbd1fbd46afce changed our COMPRESSION options, only `gz`/`bz2` ever worked.
@lub  Can you provide a comment on this issue since you are the author of that commit?
Do you remember why you submitted [that commit](https://github.com/Mailu/Mailu/commit/2316ef1162d400fc486ad72df62fbd1fbd46afce)?

This is the linked [PR1694](https://github.com/Mailu/Mailu/pull/1694). 
@nextgens Do you remember if you tested with zstd/lz4 at that time? I know it is a long time ago, but maybe you still remember?

I don't remember :$

Do we want to fix it or drop the feature?
I'd say drop the feature. I'd like to keep using the alpine dovecot package. There is no need to make our builds even more complicated and introduce more maintenance overhead. 
I'm so sad. I really liked `zstd` compression :(

In any case, you should update the configuration generator, because it confused me a lot...
@Diman0 Honestly can't remember if I tested it, but I basically made the PR to bring mailu in line with the algorithms mentioned in the dovecot documentation. I wasn't aware that they are not included in the alpine packages.

Sorry for the trouble.
On March 13th Alpine Linux got support for `zstd`compression in `dovecot` with 2.3.20-r7 which is in 3.17.
https://gitlab.alpinelinux.org/alpine/aports/-/commit/db057b7badb30307a5c66dead3049008de2c2c07
https://gitlab.alpinelinux.org/alpine/aports/-/issues/14712
https://build.alpinelinux.org/buildlogs/build-3-17-x86_64/main/dovecot/dovecot-2.3.20-r7.log

This package version is in Mailu 2.0 and thus `COMPRESSION=zstd` can now be used.

Resolving patch:
diff --git a/core/dovecot/start.py b/core/dovecot/start.py
--- a/core/dovecot/start.py
+++ b/core/dovecot/start.py
@@ -9,7 +9,7 @@
 from podop import run_server
 from socrate import system, conf
 
-system.set_env(log_filters=r'waitpid\(\) returned unknown PID \d+$')
+system.set_env(log_filters=r'Error\: SSL context initialization failed, disabling SSL\: Can\'t load SSL certificate \(ssl_cert setting\)\: The certificate is empty$')
 
 def start_podop():
     system.drop_privs_to('mail')
diff --git a/core/postfix/start.py b/core/postfix/start.py
--- a/core/postfix/start.py
+++ b/core/postfix/start.py
@@ -11,7 +11,6 @@
 from socrate import system, conf
 
 system.set_env(log_filters=[
-    r'the Postfix mail system is running\: \d+$',
     r'(dis)?connect from localhost\[(\:\:1|127\.0\.0\.1)\]( quit=1 commands=1)?$',
     r'haproxy read\: short protocol header\: QUIT$',
     r'discarding EHLO keywords\: PIPELINING$',


Test patch: