inukshuk/latex-decode — comment on commit 819e666ce00c4f6e8d8175bec9aae540fc0b9f18
File:  line 0
By alexeymuranov:

Just an idea (i will think if i can make it more clear): i think that the proper way to deal with any latex source would be to define some kind of `tokenize` method to pass it through, rather than using `gsub` with regular expressions. The TeXbook defines precisely how a TeX source is tokenized, and i think (but i am not sure) that LaTeX uses the same rules. Of course it is quite complex and depends on the current category of each input symbol, but for usual TeX documents, which are not the source of some style or macro, it can be assumed that symbol categories are fixed.