alphagov/smartdown — pull request #105 by FriedSock (2014-11-28 10:38:43+00:00)
Title: [DEFINITELY DO NOT MERGE] Handling whitespace

This commit is to show the problems / possible solutions to Example 2 on the Handling Whitespace ticket.

https://www.agileplannerapp.com/boards/105200/cards/8759

It would be simple to just ignore multiple blanklines - but then that isn't really a problem for the parser to break if they have no semantics that effect the style of the question/outcome. If we did want to catch blanklines - and allow them to be used for styling. So if we wanted to do this, here are my findings:

"# Lovely title\n\n\nline of content\n\n\nanotherlineofcontent"

This line is an example of how we would like to be able to write some smartdown - we have multiple blank lines separating our markdown paragraphs that will be actually be recognised.

it is completely possible to parse the above line like this:

{:body=>[
{:h1=>"Lovely title"@2},
{:blanklines=>"\n"@16},
{:p=>"line of content\n"@17},
{:blanklines=>"\n"@34},
{:p=>"anotherlineofcontent"@35}
]
}

the blanklines nodes could then be converted to markdown paragraphs
themselves, just like paragraphs.

However, there appears to be a bug in parselet. After defining a
grammar to do this, the following line:

"# Lovely title\n\nline of content\n\n\nanotherlineofcontent"

will parse incorrectly as:

{:body=>[
{:h1=>"Lovely title"@2},
{:p=>"line of content\n"@16}
]
}

What appears to be happening is: Because there are no blanklines to be matched between the first 2 lines, when ever any blank lines (or anything else in the grouped match) are matched subsequently: the labelling is being swallowed somehow, because parsing didn't fail, the AST being returned is just wrong.

However, there is a weird possible fix without delving deep into parselet and then hoping a PR gets merged. We can add intermediate labelling that doesn't get swallowed, so we will get an AST like this:

{:body=>[
{:h1=>"Lovely title"@2},
{:block=>{:p=>"line of content\n"@16}},
{:block=>[
{:blanklines=>"\n"@33},
{:p=>"anotherlineofcontent"@34}
]
}
]
}

So we could have `block` nodes in our AST that will contain an element, an an optional blankline. We could then just flatten this in the Transformer.