Repository: getpelican/pelican

Issue:
LOG_FILTER not working as it should?
Good evening folks,

I have a problem with LOG_FILTER. According to the docs, one can remove `TAG_SAVE_AS is set to False` by configuring `LOG_FILTER` this way:

```python
import logging
LOG_FILTER = [(logging.WARN, 'TAG_SAVE_AS is set to False')]
```

So to ignore `AUTHOR_SAVE_AS is set to False` and `CATEGORY_SAVE_AS is set to False`, it must be :

```python
import logging
LOG_FILTER = [
    (logging.WARN, 'AUTHOR_SAVE_AS is set to False'),
    (logging.WARN, 'CATEGORY_SAVE_AS is set to False')
]
```
Right?

So this is what I did:
```bash
$ head pelicanconf.py -n 20
#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals
import datetime
import logging

import pelican

PELICAN_VERSION = pelican.__version__

[...]

LOG_FILTER = [
    (logging.WARN, 'AUTHOR_SAVE_AS is set to False'),
    (logging.WARN, 'CATEGORY_SAVE_AS is set to False')
]

AUTHOR_SAVE_AS = False
AUTHORS_SAVE_AS = False
TAG_SAVE_AS = False
TAGS_SAVE_AS = False
CATEGORY_SAVE_AS = False
CATEGORIES_SAVE_AS = False
ARCHIVES_SAVE_AS = False

$ make publish
pelican /mnt/c/Users/max/Code/mlcdf/content -o /mnt/c/Users/max/Code/mlcdf/output -s /mnt/c/Users/max/Code/mlcdf/publishconf.py
WARNING: CATEGORY_SAVE_AS is set to False
WARNING: AUTHOR_SAVE_AS is set to False
Done: Processed 2 articles, 0 drafts, 4 pages, 1 hidden page and 0 draft pages in 0.52 seconds.
```
2 things here:
- it didn't work: `AUTHOR_SAVE_AS is set to False` and `CATEGORY_SAVE_AS is set to False` are still logged.
- and, you may have noticed that, despite setting `TAG_SAVE_AS = False`, and not filtering it, I did not see `TAG_SAVE_AS is set to False` in the logs.

Using the template to filter worked (but it will also filter out other logs matching the template which is not what you would necessary want):
```
import logging
LOG_FILTER = [
    (logging.WARN, '%s is set to %s'),
]
```

* Am I missing something?
* What's the intended behaviour here? Is there an actual bug in the code? Or is the code example in the docs just wrong?

I'm a bit lost. 🙃 

Maxime


Discussion:
I checked the code and you're correct. Any templated log has to be filtered with the template, not individually. This needs fixing.

Also, I don't get why `TAG_SAVE_AS is set to False` is not appearing. It should be.
Wrote a PR that will fix the issue.

IMHO `TAG_SAVE_AS is set to False` is not showing because you don't have any tag in the content frontmatter.

> IMHO `TAG_SAVE_AS is set to False` is not showing because you don't have any tag in the content frontmatter.

Well, this makes sense. Thanks!



Resolving patch:
diff --git a/pelican/log.py b/pelican/log.py
--- a/pelican/log.py
+++ b/pelican/log.py
@@ -110,11 +110,13 @@ def filter(self, record):
         else:
             self._raised_messages.add(message_key)
 
-        # ignore LOG_FILTER records by templates when "debug" isn't enabled
+        # ignore LOG_FILTER records by templates or messages
+        # when "debug" isn't enabled
         logger_level = logging.getLogger().getEffectiveLevel()
         if logger_level > logging.DEBUG:
-            ignore_key = (record.levelno, record.msg)
-            if ignore_key in self._ignore:
+            template_key = (record.levelno, record.msg)
+            message_key = (record.levelno, record.getMessage())
+            if (template_key in self._ignore or message_key in self._ignore):
                 return False
 
         # check if we went over threshold


Test patch:
diff --git a/pelican/tests/test_log.py b/pelican/tests/test_log.py
new file mode 100644
--- /dev/null
+++ b/pelican/tests/test_log.py
@@ -0,0 +1,78 @@
+import logging
+import unittest
+from collections import defaultdict
+
+from pelican import log
+from pelican.tests.support import LogCountHandler
+
+
+class TestLog(unittest.TestCase):
+    def setUp(self):
+        super().setUp()
+        self.logger = logging.getLogger(__name__)
+        self.handler = LogCountHandler()
+        self.logger.addHandler(self.handler)
+
+    def tearDown(self):
+        self._reset_limit_filter()
+        self.logger.removeHandler(self.handler)
+        super().tearDown()
+
+    def _reset_limit_filter(self):
+        log.LimitFilter._ignore = set()
+        log.LimitFilter._raised_messages = set()
+        log.LimitFilter._threshold = 5
+        log.LimitFilter._group_count = defaultdict(int)
+
+    def test_log_filter(self):
+        def do_logging():
+            for i in range(5):
+                self.logger.warning('Log %s', i)
+                self.logger.warning('Another log %s', i)
+        # no filter
+        do_logging()
+        self.assertEqual(
+            self.handler.count_logs('Log \\d', logging.WARNING),
+            5)
+        self.assertEqual(
+            self.handler.count_logs('Another log \\d', logging.WARNING),
+            5)
+        self.handler.flush()
+        self._reset_limit_filter()
+
+        # filter by template
+        log.LimitFilter._ignore.add((logging.WARNING, 'Log %s'))
+        do_logging()
+        self.assertEqual(
+            self.handler.count_logs('Log \\d', logging.WARNING),
+            0)
+        self.assertEqual(
+            self.handler.count_logs('Another log \\d', logging.WARNING),
+            5)
+        self.handler.flush()
+        self._reset_limit_filter()
+
+        # filter by exact message
+        log.LimitFilter._ignore.add((logging.WARNING, 'Log 3'))
+        do_logging()
+        self.assertEqual(
+            self.handler.count_logs('Log \\d', logging.WARNING),
+            4)
+        self.assertEqual(
+            self.handler.count_logs('Another log \\d', logging.WARNING),
+            5)
+        self.handler.flush()
+        self._reset_limit_filter()
+
+        # filter by both
+        log.LimitFilter._ignore.add((logging.WARNING, 'Log 3'))
+        log.LimitFilter._ignore.add((logging.WARNING, 'Another log %s'))
+        do_logging()
+        self.assertEqual(
+            self.handler.count_logs('Log \\d', logging.WARNING),
+            4)
+        self.assertEqual(
+            self.handler.count_logs('Another log \\d', logging.WARNING),
+            0)
+        self.handler.flush()
+        self._reset_limit_filter()