tastejs/PropertyCross — pull request #299 by BobDickinson (2014-11-20 03:21:14+00:00)
Title: Implemented logical lines of code in stats

OK, I understand that this may be a somewhat controversial change, so I'm throwing this out there in case you do want to incorporate it.

In evaluating the stats produced by stats.js, we also wanted to look at total lines of code per solution. We realize that this is not a particularly useful metric, but considering that there is almost an order of magnitude difference in the range across these solutions, we wanted to see it. As we analyzed the results, we found that they were skewed fairly heavily by non-statement code (comments, blank lines, block/element framing lines, etc). If you consider an implementation using CoffeeScript (which is very concise in terms of vertical space for a unit of logic) versus the various C-like languages (including Java and JavaScript), which can be slightly to significantly larger vertically for a given unit of logic depending on how braces are positioned, you will see that counting raw lines does not really do a good job at estimating relative complexity (particularly consider variations in comments and blank lines between the solutions).

To do any kind of proper complexity analysis would be a fairly huge job (particularly given the wide variety of languages used, and the tools that would be required per-language). But it did seem like some attempt to estimate logical lines of code would produce a somewhat better gross evaluation of complexity for comparison purposes than raw lines of code. After looking at some research in the area, we came up with the following mechanism:

Remove block comments
Remove blank lines (including those created in the previous step)
Remove lines that contain only line comments
Remove lines that contain only block/element framing ({, }, [, ] in C-like, "end" in Ruby, etc)

This produces a logical lines of code (LLOC) count representing only lines that include program logic. In looking at the results against the PropertyCross implementations, we felt that the LLOC metrics derived using this method were a more fair and equitable representation of general complexity of the solutions than the raw/physical line of code count currently used.

By default, stats.js functions as before, with the exception that the JSON output (if specified as a command line argument) now includes the line count for each solution.

If the argument "lloc" is passed in, the new LLOC metric will be used instead of the raw/physical line counting. If, additionally, the argument "verify" is passed in, an lloc file will be created for every source file evaluation (so that you can compare the source file to the lloc file to determine what was removed in lloc calculation). The lloc files will have the same name, including extension, as the source file, but with ".lloc" appended.

We diffed a significant number of source files and lloc files for each solution type in order to review the algorithm performance.