getpelican/pelican — pull request #1030 by Rogdham (2013-08-10 15:22:08+00:00)
Title: Limit and filter logs

Having clear logs messages is very important to be able to spot quickly when something is wrong. In other words, if you have too many logs messages for nothing, you won't notice the very one important log message in the middle of all the others.
So to limit the number of log messages (especially the useless ones), three ideas have been implemented: removing duplicates, limit the number of similar log messages, and allow the user to specify some log messages she wants to filter out.

More precisely, this PR does 4 different things:
1. drop duplicates logs (up to WARN level)
2. allow for logs to be grouped in categories and enforce a maximum number of logs per group (5 lines of log)
3. change previously existing logs to take advantage of the previous points (e.g. almost revert c875c27e83941ecd2bcd401efe07b35421de704c, which simplyfies the code base)
4. add a `LOG_FILTER` setting

Points 2 and 4 are explained below.

Thanks for @saimn for clarifying commit c875c27e83941ecd2bcd401efe07b35421de704c, and people on IRC in general to vote for the name of the new setting, `LOG_FILTER`.

# Log grouping

This feature was introduced for empty `alt` attributes only in #895. The idea was that if you have 1000 images, instead of having 1000 times `WARNING: Empty alt attribute for image [image_src] in [file_source]`, you only have a few, and then a message saying that there is more.

Here is an example after this PR (as you can see, the threshold has been set to 5):

WARNING: Unable to find 1.png, skipping url replacement
WARNING: Unable to find 2.png, skipping url replacement
WARNING: Unable to find 3.png, skipping url replacement
WARNING: Unable to find 4.png, skipping url replacement
WARNING: Other ressources were not found and their urls not replaced

To use the log grouping function, instead of passing the log message, give a tuple `(log_message, message_if_too_many)`.

For example, change:

logger.warning("Unable to find {fn}, skipping url replacement".format(fn=value))

for:

logger.warning(("Unable to find {fn}, skipping url replacement".format(fn=value),
"Other ressources were not found and their urls not replaced"))

# Add a `LOG_FILTER` setting

The `LOG_FILTER` setting is added. It should contain an iterable of tuples of the form `(log_level, log_message)`.

The idea behind this setting is that some configurations raise useless logs.

Say for example that you don't want to generate tags pages on you website. You will get this warning every time for nothing, as you did set this setting to `False` on purpose:

WARNING: TAG_SAVE_AS is set to False

So now, you simply adds that to you setting file:

import logging
LOG_FILTER = [(logging.WARN, 'TAG_SAVE_AS is set to False)] # or replace the list by a tuple/…

That means that you need one line per log that you want to remove… but this should not be a problem. In other words, if you want to remove a lots of logs, you are doing something wrong in the first place.

# Putting everything together

Say you did not pay attention and did not specify `alt` attributes (that bad!). Here are the warnings you get:

WARNING: Empty alt attribute for image 1.png in /tmp/blog/content/post1.md
WARNING: Empty alt attribute for image 2.png in /tmp/blog/content/post1.md
WARNING: Empty alt attribute for image hbar.png in /tmp/blog/content/post1.md
WARNING: Empty alt attribute for image 3.png in /tmp/blog/content/post1.md
WARNING: Other images have empty alt attributes

But wait, the *hbar.png* file is just an horizontal bar without meaning, so it's fine **(*)** if it has an empty `alt` attribute! So you add that warning to `LOG_FILTER` in your setting file, and generate your blog again. This time, you get:

WARNING: Empty alt attribute for image 1.png in /tmp/blog/content/post1.md
WARNING: Empty alt attribute for image 2.png in /tmp/blog/content/post1.md
WARNING: Empty alt attribute for image 3.png in /tmp/blog/content/post1.md
WARNING: Empty alt attribute for image 4.png in /tmp/blog/content/post1.md
WARNING: Other images have empty alt attributes

In other words, you still have the same number of warnings (you forgot to put a lot of `alt` attributes, shame on you), but now you know that the image *4.png* is missing an `alt` attribute as well.

**(*)**: well, you should probably do it in CSS instead, but that was the easiest example that I thought about for legitimate images without `alt` attributes.