Refactoring parts of the modeline parsing

Mostly splitting finding out if it's a modeline from chk_modeline into another
function. Also some more documentation, formatting and sanity checks.