support nested brackets and spaces in links

- behavior matches sundown

depends on #397 and #398