fishman/ctags — pull request #162 by b4n (2014-12-02 17:08:09+00:00)
Title: Escape control control characters when writing tag fields

**Warning: *Do NOT merge this just yet.*** *It is currently more of an issue with sample code than a real PR.*

What's wrong?
---
Nothing prevents the various tag fields (from tag name to tag signature through file name) from containing arbitrary data, including control characters like `\t` or `\n`. The presence of most of these will break the output, especially `\t` and `\n` which are obviously used as part of the tag file structure.

Some parsers, like [the JavaScript one](/fishman/ctags/blob/master/jscript.c#L276), have limited handling of such input to mitigate possible problems, but this is often not enough (it for example won't help with file names) and is fragile as each parser has to do it manually.

What can we do?
---
I see two possible approaches:

1. Filter the offending bytes when creating the tag entry, in
`makeTagEntry()`. I see two implications worth noting with this
approach:
1. It requires string duplication that may be avoided with other
solutions.
2. It is an arbitrary limitation of the in-memory format, which may
or may not be wanted (e.g. when making CTags a library).
2. Filter the offending bytes when writing the tag entry. This has the
advantage of only affecting the output format (the very goal of the
change) and not the internal representation, so it has very low
implication on how internals handle these.

This patch attached is a mostly working attempt at the second approach. It isn't completely ready yet because it doesn't handle `%Ns` and the like that are use in 2 places (see the `FIXME`), but is a proof of concept.

The implementation here might *not* be the one we want to go with because it basically re-implements `fprintf()`, which is a non-trivial function for non-trivial cases: e.g. support for `%-Ns` is not trivial.

The advantage is that it requires minimal changes on how the output is generated, and that the escaping being implicit lowers a lot the risk of forgetting to escape some parts. It also avoids having to duplicate strings to build the escaped version, as filtering can be done straight when producing the actual output in most cases.

Seriously?
---
For those who wonder how these can actually happen, there are two easy ways:

* Parse a file with e.g. a `\n` in the name:

```shell
bash$ cat > $'/tmp/test\n.c' <<<'int main(void){}'
bash$ ctags -o - /tmp/test*.c
```
…and you'll see that the output is pretty much garbage as not only there's an extra `\n` but it also broke sorting.
* Parse a file that generates a tag with such a control character in it. Simpler way I know is through a property in JavaScript or a `define()` in PHP, but I guess there might be many more:

```shell
bash$ cat > /tmp/test.php <<<$'