tanoku/redcarpet — pull request #101 by lsegal (2012-01-15 22:32:47+00:00)
Title: Accept arbitrary encodings, not just US-ASCII/UTF-8

First I should point out that this issue is related to issues #89 (and a patch in #95). However, I don't believe #89 solves the issue properly, rather, it hides the underlying issue.

The issue here is that Markdown is actually not a UTF-8 specific markup format. The fact that redcarpet requires UTF-8 "breaks" the specification, IMO, which makes no mention that Markdown input has to be in UTF-8 format. Specifically, it is completely valid for a user to require their input to be in, say, ISO-8859-1 (a subset of ASCII-8BIT), which is ASCII compatible, but *not UTF-8 compatible*. A functioning markdown library should be able to parse any ASCII compatible encoding, since all Markdown syntax is specified in the ASCII range. In other words, there should *not* be a UTF-8 requirement on Markdown input.

This pull request removes the UTF-8 input check, thereby allowing input to be in any ASCII compatible format (which is every encoding I know of including even encodings like SJIS). It seems much wiser to me to specify encoding requirements in documentation, not at runtime, as runtime checks might needlessly restrict the input when it is perfectly valid.

This patch has 2 commits, the first commit fixes a broken test from #95. The reason the test is broken is mentioned in the comments on the commit in the referenced pull request. After all encoding behaviour is working as expected, namely, redcarpet returns the data in the original encoding, I removed the UTF-8/US-ASCII checks and added a new test which contains non-UTF-8 data (specifically binary encoding). The data is properly parsed by the library and properly returned in the original encoding, so it would seem to me that there are no real issues with this functionality.

Note that there is still one underlying issue in rc_render.c where buf2str() is creating strings from buf structs but has no access to any encoding information, as it is coming from callbacks in sundown. In this case, I return UTF-8. This should probably be fixed, but I imagine this would require sundown to pass encoding information in the callbacks (via the buf struct). Perhaps this can be addressed in a future commit. Until then, I was unable to produce a failing test case using the code in my pull request, probably because I don't know exactly how these render callbacks are being used. If you can help produce a failing commit, I would be glad to make this code encoding aware too.