Repository: fail2ban/fail2ban

Issue:
dovecot filter - Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'
I am using Centos 7, with fail2ban 0.9.1 and firewalld 0.3.9
I use the firewallcmd-new action for banning / unbanning.

I just restarted fail2ban and get the following:

2015-01-09 10:26:33,713 fail2ban.server         [18465]: INFO    Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.1
2015-01-09 10:26:33,715 fail2ban.database       [18465]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2015-01-09 10:26:33,727 fail2ban.jail           [18465]: INFO    Creating new jail 'sshd'
2015-01-09 10:26:33,733 fail2ban.jail           [18465]: INFO    Jail 'sshd' uses poller
2015-01-09 10:26:33,747 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:33,748 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,117 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/secure
2015-01-09 10:26:34,117 fail2ban.filter         [18465]: INFO    Set maxRetry = 5
2015-01-09 10:26:34,118 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,118 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,119 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,119 fail2ban.filter         [18465]: INFO    Set maxlines = 10
2015-01-09 10:26:34,161 fail2ban.server         [18465]: INFO    Jail sshd is not a JournalFilter instance
2015-01-09 10:26:34,165 fail2ban.jail           [18465]: INFO    Creating new jail 'apache-noscript'
2015-01-09 10:26:34,166 fail2ban.jail           [18465]: INFO    Jail 'apache-noscript' uses poller
2015-01-09 10:26:34,167 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,167 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,170 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/httpd/ssl_error_log
2015-01-09 10:26:34,172 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/httpd/error_log
2015-01-09 10:26:34,172 fail2ban.filter         [18465]: INFO    Set maxRetry = 6
2015-01-09 10:26:34,173 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,173 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,174 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,181 fail2ban.jail           [18465]: INFO    Creating new jail 'apache-overflows'
2015-01-09 10:26:34,182 fail2ban.jail           [18465]: INFO    Jail 'apache-overflows' uses poller
2015-01-09 10:26:34,182 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,183 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,185 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/httpd/ssl_error_log
2015-01-09 10:26:34,187 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/httpd/error_log
2015-01-09 10:26:34,188 fail2ban.filter         [18465]: INFO    Set maxRetry = 2
2015-01-09 10:26:34,189 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,189 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,190 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,198 fail2ban.jail           [18465]: INFO    Creating new jail 'apache-botsearch'
2015-01-09 10:26:34,199 fail2ban.jail           [18465]: INFO    Jail 'apache-botsearch' uses poller
2015-01-09 10:26:34,200 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,200 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,203 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/httpd/ssl_error_log
2015-01-09 10:26:34,205 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/httpd/error_log
2015-01-09 10:26:34,205 fail2ban.filter         [18465]: INFO    Set maxRetry = 2
2015-01-09 10:26:34,206 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,207 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,208 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,217 fail2ban.jail           [18465]: INFO    Creating new jail 'vsftpd'
2015-01-09 10:26:34,218 fail2ban.jail           [18465]: INFO    Jail 'vsftpd' uses poller
2015-01-09 10:26:34,219 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,219 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,229 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/vsftpd.log
2015-01-09 10:26:34,230 fail2ban.filter         [18465]: INFO    Set maxRetry = 5
2015-01-09 10:26:34,231 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,231 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,232 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,241 fail2ban.jail           [18465]: INFO    Creating new jail 'postfix'
2015-01-09 10:26:34,241 fail2ban.jail           [18465]: INFO    Jail 'postfix' uses poller
2015-01-09 10:26:34,242 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,242 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,245 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/maillog
2015-01-09 10:26:34,246 fail2ban.filter         [18465]: INFO    Set maxRetry = 5
2015-01-09 10:26:34,247 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,247 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,248 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,261 fail2ban.server         [18465]: INFO    Jail postfix is not a JournalFilter instance
2015-01-09 10:26:34,265 fail2ban.jail           [18465]: INFO    Creating new jail 'dovecot'
2015-01-09 10:26:34,266 fail2ban.jail           [18465]: INFO    Jail 'dovecot' uses poller
2015-01-09 10:26:34,267 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,267 fail2ban.jail           [18465]: INFO    Initiated 'polling' backend
2015-01-09 10:26:34,277 fail2ban.filter         [18465]: INFO    Added logfile = /var/log/maillog
2015-01-09 10:26:34,277 fail2ban.filter         [18465]: INFO    Set maxRetry = 5
2015-01-09 10:26:34,278 fail2ban.filter         [18465]: INFO    Set jail log file encoding to ANSI_X3.4-1968
2015-01-09 10:26:34,279 fail2ban.actions        [18465]: INFO    Set banTime = 600
2015-01-09 10:26:34,280 fail2ban.filter         [18465]: INFO    Set findtime = 600
2015-01-09 10:26:34,295 fail2ban.server         [18465]: INFO    Jail dovecot is not a JournalFilter instance
2015-01-09 10:26:34,304 fail2ban.jail           [18465]: INFO    Jail 'sshd' started
2015-01-09 10:26:34,324 fail2ban.jail           [18465]: INFO    Jail 'apache-noscript' started
2015-01-09 10:26:34,362 fail2ban.jail           [18465]: INFO    Jail 'apache-overflows' started
2015-01-09 10:26:34,385 fail2ban.jail           [18465]: INFO    Jail 'apache-botsearch' started
2015-01-09 10:26:34,403 fail2ban.jail           [18465]: INFO    Jail 'vsftpd' started
2015-01-09 10:26:34,417 fail2ban.jail           [18465]: INFO    Jail 'postfix' started
2015-01-09 10:26:34,429 fail2ban.jail           [18465]: INFO    Jail 'dovecot' started
2015-01-09 10:27:02,975 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:13 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t1\tuser=dise\xf1o\n'
2015-01-09 10:27:02,976 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:20 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t2\tuser=dise\xf1o\n'
2015-01-09 10:27:02,977 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:30 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t3\tuser=dise\xf1o\n'
2015-01-09 10:27:02,979 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:52 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t8\tuser=dise\xf1o\n'
2015-01-09 10:27:02,991 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:58 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t9\tuser=dise\xf1o\n'
2015-01-09 10:27:02,991 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:54:08 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t10\tuser=dise\xf1o\n'
2015-01-09 10:27:02,994 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:54:30 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t16\tuser=dise\xf1o\n'
2015-01-09 10:27:03,580 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:13 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t1\tuser=dise\xf1o\n'
2015-01-09 10:27:03,597 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:20 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t2\tuser=dise\xf1o\n'
2015-01-09 10:27:03,609 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:30 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t3\tuser=dise\xf1o\n'
2015-01-09 10:27:03,632 fail2ban.filter         [18465]: WARNING Error decoding line from '/var/log/maillog' with 'ANSI_X3.4-1968'. Continuing  to process line ignoring invalid characters: 'Jan  7 14:53:52 bimomotor dovecot: auth: Debug: client passdb out: FAIL\t8\tuser=dise\xf1o\n'

The entries in the maillog from 14:53:52 show:

Jan  7 14:53:52 bimomotor dovecot: auth: Debug: client passdb out: FAIL 8   user=diseño
Jan  7 14:53:52 bimomotor postfix/smtpd[17078]: warning: unknown[212.185.111.115]: SASL LOGIN authentication failed: Connection lost to authentication server
Jan  7 14:53:52 bimomotor dovecot: auth: Debug: client in: AUTH 9   LOGIN   service=smtp    nologin lip=5.134.116.130   rip=212.185.111.115 resp=ZGlzZfFv (previous base64 data may contain sensitive data)

It seems odd that this cannot be parsed...



Discussion:
Is just a warning, to prevent it ...
Just add `logencoding = utf-8` to jail `dovecot`. 
Or if all your log files are sure in utf-8, you can replace `auto` with `utf-8` in default:

``` bash
[DEFAULT]
...
# logencoding = auto
logencoding = utf-8
...
```

indeed... but to prevent such reports in the future -- may be we should either add such information straight into the WARNING Message or may be even initiate some wiki page listing those and so we could refer /url them from msgs for more detail? 

I thought, the message "Error decoding line from '...' with 'ANSI_X3.4-1968'" is enough meaningful...
But perhaps we should change a message to "Invalid encoding 'ANSI...' for line from '...'".
Imho, it is more speaking...

By the way, I think, set the `auto` to log encoding can be possibly a security "leak", because If someone give as user name a unicode characters, and fail2ban by start can incorrectly recognise encoding as `unicode` - the parsing of this log in the future will never recognize any failure up to next start / log-rotate. So I always set it explicit to real encoding, used from corresponding service (most of the time is `utf-8`).


Resolving patch:
diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py
--- a/fail2ban/server/filter.py
+++ b/fail2ban/server/filter.py
@@ -797,7 +797,9 @@ def readline(self):
 			line = line.decode(self.getEncoding(), 'strict')
 		except UnicodeDecodeError:
 			logSys.warning(
-				"Error decoding line from '%s' with '%s'. Continuing "
+				"Error decoding line from '%s' with '%s'."
+				" Consider setting logencoding=utf-8 (or another appropriate"
+				" encoding) for this jail. Continuing"
 				" to process line ignoring invalid characters: %r" %
 				(self.getFileName(), self.getEncoding(), line))
 			if sys.version_info >= (3,): # In python3, must be decoded


Test patch: