hoedown/hoedown — pull request #121 by jmendeth (2014-09-05 19:16:19+00:00)
Title: [Experiment] Direct source mapping

**Important:** The code is just a proof of concept and will not be merged. This PR is for feedback.

This demonstrates how, with less than 100 lines of infrastructure, Hoedown can be able to track the point in source a snippet came from, reliably, and with minimal overhead.

Please read each commit in order; the description explains what's going on in each step.
Interesting steps are 2 & 3 (implement source mapping) and step 4 (use it to locate math spans).

See it **[in action](https://cloud.githubusercontent.com/assets/1177304/4169632/52a4743c-3529-11e4-9460-8fa66fd417bd.png)**!

## The API

As @zdne said in #28 (and I now agree), a block in Markdown isn't necessarily located within a single, *contiguous* range of characters in the original source. Here, instead of passing around a `src_map` object and creating a set of methods to interact with it, I've chosen a lower-level approach.

There's a **single method**, `hoedown_document_locate`, that is passed a pointer to a character. The method will try to deduce the position of a character in the Markdown source. Some characters don't have an associated position, creating "gaps" instead of a contiguous range.

## The concept

A stack of **mappings** is maintained. Each mapping associates a chunk of data in memory, with its position in the source.

When functions extract parts of the source and pass it to the callback, they do it like so:

~~~ c
hoedown_buffer language = { 0, 0, 0, 0, 0, 0 };
// ...
language.data = data + lang_beg;
language.size = lang_end - lang_beg;
// ...
doc->md.fenced_code(ob, &text, &language, doc->md.opaque);
~~~

Even though the language is passed through a `hoedown_buffer`, it still points to the *same memory area*, so we can easily deduce its position in source by looking at the mapping stack.

## Limitations

First of all, this tracks the position **in the preprocessed source**. So if the input contains tabs, carriage returns, link or footnote references, the preprocessed source will be different than the input and it won't work. This can be worked around but it's not trivial.

It's **direct** source mapping. It only tracks things that were extracted verbatim from the input. So it'll work on code, math, and normal text, but it won't work with link URLs, because link URLs are unescaped instead of copied directly. It'd suffice for cases like uranusjr/macdown#121 though.

However, direct source mapping is the base for implementing more sophisticated mapping later.