fishman/ctags — pull request #135 by b4n (2014-11-23 17:51:34+00:00)
Title: javascript: Import Geany changes and other fixes

This PR imports the JavaScript changes from Geany (d272d11740ea476f0d415b81ecf80eeb9d98e027, 7064754788720ba16822e40a53dd0697aae410fa, 34fccae69934f6200c132ecf359f73e6cb36e418, 5243ab7b2c35478ea68b5671edd01d178cb316b4, 1b96135186ba273b223a5777e23d0e9ba3a0789b, 5a08f161325383e990d9c5d6d7d0f33bec958232, 773d2ba385982b3757d6c6910e113c1d18d7d0d2, 0ac9cf7ab204a7e87a0de9e888df179e571aeafc) and adds other fixes and improvements.

Most noteworthy changes are fixes for various unterminated statements (statements without a terminating semicolon), and report for function/method/class' signature.

Note however that there still are quite a few constructs where the parser will fail to generate the correct tags, but that's better now.
The main reason why this parser won't ever get everything completely right is that JavaScript is a very subtle language that basically requires to understand every grammar to be able to even delimit statements (because of the complex optional semicolons rules). Another reason is that this parser is a mess that grew in all directions so no one really understands it (some parts are clearly old and irrelevant now but weren't removed -- 1d7e331d92cad27310a7ff724c38379d64826fc1 is a good example, and it's not alone). This leads most modifications to have unexpected impact on mostly unrelated things because the parser worked only by relying on some more or less lucky state, basically forcing one to fix ten more bugs not for a fix to trigger them.

Also, the current test cases are not extensive enough to catch everything that could possibly break (actually, nothing really is as no one really knows what could happen). To avoid breaking too many things I checked manually the difference on the generated output for all JavaScript on my system (which includes among other things Mozilla and Chromium extensions, jQuery, jQuery UI, Prototype, Bootstrap, PHPMyAdmin, various web pages, WordPress, WebKit, some Cinnamon and GNOME-Shell sources and various others). This is useful as they are real-life extensive JavaScript with the various quirks, but it is *huge* as you would imagine (some files are several tens of thousands of lines, like *pdf.js*, *jit.js*, *env.js* or *firebug* for example), and it doesn't really make sense to include them in the CTags test suite (or does it?), as not only they are huge and don't test one particular thing, but also the parser doesn't yet (and probably will never) provide perfect results, so the *expected.tags* would basically have to be changed (I didn't write *fixed*) for every commit -- the only advantage would be that hopefully one would check if the difference is a regression or not (and in case of a regression whether it's an important one or a small detail that wasn't right anyway).
So, my point is that we cannot do much about it, but people touching this parser probably should check the results against various huge real-world examples as I did to mitigate breakage.