Repository: fail2ban/fail2ban

Issue:
Parse audit.log (SElinux)
Parsing audit.log from SElinux might be an alternative to secure.log. As mentioned in #289, audit.log can't currently be parsed because unix timestamps aren't at beginning of lines.
I'm not sure if multi-line regex will have any effect on this but currently parsing can work if I remove match to beginning of a line from Epoch matching function (`self.setRegex("^\d{10}(\.\d{6})?")`) in `datetemplate.py`.

Sample log (as diverse I'm able to dig):

```
type=USER_LOGIN msg=audit(1373330716.415:4063): user pid=11998 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=173.242.116.187 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1373330717.441:4068): user pid=12000 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct=28756E6B6E6F776E207573657229 exe="/usr/sbin/sshd" hostname=? addr=173.242.116.187 terminal=ssh res=failed'
type=USER_ERR msg=audit(1373330717.575:4070): user pid=12000 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident acct="?" exe="/usr/sbin/sshd" hostname=173.242.116.187 addr=173.242.116.187 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1373330717.576:4073): user pid=12000 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct=28696E76616C6964207573657229 exe="/usr/sbin/sshd" hostname=? addr=173.242.116.187 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1372546928.726:52008): user pid=21569 uid=0 auid=0 ses=76 subj=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="sshd" exe="/usr/sbin/sshd" hostname=? addr=113.240.248.18 terminal=ssh res=failed'
type=USER_ERR msg=audit(1372557500.401:61747): user pid=23684 uid=0 auid=0 ses=76 subj=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident acct="?" exe="/usr/sbin/sshd" hostname=113.240.248.18 addr=113.240.248.18 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1372557500.402:61750): user pid=23684 uid=0 auid=0 ses=76 subj=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct=28696E76616C6964207573657229 exe="/usr/sbin/sshd" hostname=? addr=113.240.248.18 terminal=ssh res=failed'
type=USER_AUTH msg=audit(1373129280.772:9): user pid=1277 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="root" exe="/usr/sbin/sshd" hostname=? addr=194.228.20.113 terminal=ssh res=failed'
```



Discussion:
date formats are discussed #214 so after that's done we just need to make a filter for this.

@kwirk , thoughts on how to handle this? a &lt;date&gt; in a filter expression which defaults at the start?

I'm not sure what the best way to resolve this is. 
- One option is to remove the anchor from "Epoch" filter, but this may have unexpected effects on other lines which happen to have 10 concurrent digits.
- The custom date template changes in _0.9_ could be tweaked to allow lone use of `%s` format, which actually creates Epoch template. Then a date pattern of `^type=USER_ERR msg=audit(%s:`. A bit hacky, but would work.
- A new datetemplate could be added to the default detector…

---

Having wrote the above, I started off on a tangent and came up with using look behind/ahead assertions, which I believe should work safely:

``` diff
diff --git a/server/datetemplate.py b/server/datetemplate.py
index 86eeee8..1e992c1 100644
--- a/server/datetemplate.py
+++ b/server/datetemplate.py
@@ -75,8 +75,7 @@ class DateEpoch(DateTemplate):

    def __init__(self):
        DateTemplate.__init__(self)
-       # We already know the format for TAI64N
-       self.setRegex("^\d{10}(\.\d{6})?")
+       self.setRegex("(?:^|(?P<selinux>(?<=audit\()))\d{10}(?:\.\d{3,6})?(?(selinux)(?=:\d+\)))")

    def getDate(self, line):
        date = None
diff --git a/testcases/datedetectortestcase.py b/testcases/datedetectortestcase.py
index 8a83870..95d1a29 100644
--- a/testcases/datedetectortestcase.py
+++ b/testcases/datedetectortestcase.py
@@ -83,6 +83,7 @@ class DateDetectorTest(unittest.TestCase):
            "<01/23/05@21:59:59>",
            "050123 21:59:59", # MySQL
            "Jan-23-05 21:59:59", # ASSP like
+           "audit(1106513999.123:987)", # SELinux
            ):
            log = sdate + "[sshd] error: PAM: Authentication failure"
            # exclude

```

``` python
In [21]: line = 'type=USER_AUTH msg=audit(1373129280.772:9): user pid=1277 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg=\'op=pubkey acct="root" exe="/usr/sbin/sshd" hostname=? addr=194.228.20.113 terminal=ssh res=failed\''

In [22]: match = re.search("(?:^|(?P<selinux>(?<=audit\()))\d{10}(?:\.\d{3,6})?(?(selinux)(?=:\d+\)))", line)

In [23]: line[:match.start()] + line[match.end():]
Out[23]: 'type=USER_AUTH msg=audit(:9): user pid=1277 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg=\'op=pubkey acct="root" exe="/usr/sbin/sshd" hostname=? addr=194.228.20.113 terminal=ssh res=failed\''

In [24]: match.group()
Out[24]: '1373129280.772'
```

What do you think?

looks quite hacky but until there's a better way I'd rather see an selinux auditd filter as possible rather than it omitted. Cleanups can occur later if we've set the base filter for it. 

Sounds good enough to continue development

`a <date> in a filter expression which defaults at the start?`

I am thinking the same -- that in 0.9 we should remove this automagical search for a datepattern and have it specified within the failregex.

and may be only to ease transition "prepend `<date>` if none is present in the failregex (and ignoreregex)" ?

> I am thinking the same -- that in 0.9 we should remove this automagical search for a datepattern and have it specified within the failregex.

sounds good.

@kwirk  want to commit your patch now?

the majority of log files have the date at the beginning.

we could have a way to instruct a specific jail to ignore the ^ anchor for
the date?
On Jul 28, 2013 12:43 PM, "Yaroslav Halchenko" notifications@github.com
wrote:

> and may be only to ease transition "prepend <date> if none is present in
> the failregex (and ignoreregex)" ?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/fail2ban/fail2ban/issues/296#issuecomment-21685413
> .

> we could have a way to instruct a specific jail to ignore the ^ anchor for
> the date?

yes, but its a kludge. Even non beginning of line logs could match a long number that is interpreted as a Epoch date. It also makes it harder to move to a &lt;date&gt; in the regex with is closer to a rugged solution.

@kwirk 's regex will suffice for now.


Resolving patch:
diff --git a/testcases/samplestestcase.py b/testcases/samplestestcase.py
--- a/testcases/samplestestcase.py
+++ b/testcases/samplestestcase.py
@@ -61,11 +61,7 @@ def testFilter(self):
 		# Check filter exists
 		filterConf = FilterReader(name, "jail", basedir=CONFIG_DIR)
 		filterConf.read()
-		try:
-			filterConf.getOptions({})
-		except InterpolationMissingOptionError:
-			# some filters like selinux aren't complete
-			return
+		filterConf.getOptions({})
 
 		for opt in filterConf.convert():
 			if opt[2] == "addfailregex":
@@ -136,7 +132,7 @@ def testFilter(self):
 
 	return testFilter
 
-for filter_ in os.listdir(os.path.join(CONFIG_DIR, "filter.d")):
+for filter_ in filter(lambda x: not x.endswith('common.conf'), os.listdir(os.path.join(CONFIG_DIR, "filter.d"))):
 	filterName = filter_.rpartition(".")[0]
 	setattr(
 		FilterSamplesRegex,


Test patch: