karlcow/markdown-testsuite — pull request #15 by cirosantilli (2014-03-26 13:17:35+00:00)
Title: Automated tests.

This goes a long way to solve #5: now we have a script that does all the tests, all we need is find the best DOM normalizer to compare DOM trees.

**Features**

Install with:

sudo pip install -r requirements.txt

**All engines**

./run-tests.py

Sample output:

gfm |FF F F FFFF F F FFFF FF FF F F | 104.34s 102 20 19%
kramdown | FFF FF FF FF FFFFFFFFFFFFF F | 31.69s 102 23 22%
pandoc |FF F F FFFF FFFFF FF FFFFFFFFFFFFF FFFF FF FFF FFF F| 0.98s 102 41 40%
redcarpet | | 22.01s 102 0 0%

Extensions:

gfm |F | 1.17s 2 1 50%
kramdown | | 0.62s 2 0 0%
pandoc |F | 0.01s 2 1 50%
redcarpet | | 0.43s 2 0 0%

**Detailed diagnosis for a single engine**

./run-tests.py pandoc

Sample output for a single error:

# extensions/pandoc/fenced-code-block
======================================================================

```
a
```

----------------------------------- output:

a

----------------------------------- expected:

a

======================================================================

**Extras**

Configurations can be done under `config_local.py`.

**Implementation**

This is currently using [Beautiful Soup 4](http://www.crummy.com/software/BeautifulSoup/) `prettify` to normalize DOM outputs.

As far as I can tell there are two problems with BS4:

- this is not what `prettify` is intended for, but works for most things, except one: multiple spaces and newlines vs spaces. Even so, many tests pass, and all Redcarpet tests do.

- Some attributes added by several engines: e.g. header IDs are added by both Kramdown and Pandoc. This is harder to solve since the DOMs really are different, and I don't think BS4 has a way to strip only the right attributes (might not even be computable since some attributes like `href` must stay).

Still I think this is already useful on its own.

If anyone finds a better normalizer, you just have to modify **one** method: `run-tests.dom_normalize`.

**Implementation Notes**

- add utf8 tests like `tests/extensions/utf8.md` to make sure I'm not getting encodings wrong

- the third line of `tests/blockquote-multiline-1-space-end.md` ended in a UTF-8 non breakable space. If we really need that, lets make it very explicit and put on a separate tests.