github/linguist — pull request #1403 by arfon (2014-07-23 17:32:19+00:00)
Title: Benchmarking

Opening up this PR to get some :eyes: on this.

Basically this PR adds basic benchmarking to Linguist. To try it out copy the `samples` folder into `benchmark/samples` at the root level of the repo (note that these files are ignored in `.gitignore`).

Then try something like this:

`bundle exec rake benchmark:run compare=19c46da7...862bc3c33` where the `compare` string is two recent Git SHAs.

You should get output that looks like this:

```
Potential misclassifications for 19c46da7
bar.h is classified as C but C++ was expected
bcm2835.h is classified as Objective-C but C++ was expected
CsvStreamer.h is classified as Objective-C but C++ was expected
env.h is classified as C but C++ was expected
Field.h is classified as Objective-C but C++ was expected
gdsdbreader.h is classified as Objective-C but C++ was expected
key.h is classified as C but C++ was expected
libcanister.h is classified as Objective-C but C++ was expected
metrics.h is classified as Objective-C but C++ was expected
protocol-buffer.pb.h is classified as C but C++ was expected
qscicommand.h is classified as Objective-C but C++ was expected
qsciprinter.h is classified as Objective-C but C++ was expected
rpc.h is classified as Objective-C but C++ was expected
scanner.h is classified as C but C++ was expected
Types.h is classified as Objective-C but C++ was expected
utils.h is classified as C but C++ was expected
v8.h is classified as C but C++ was expected

Potential misclassifications for 862bc3c33
bar.h is classified as C but C++ was expected
bar.hpp is classified as No language but C++ was expected
bcm2835.h is classified as Objective-C but C++ was expected
CsvStreamer.h is classified as Objective-C but C++ was expected
env.h is classified as C but C++ was expected
epoll_reactor.ipp is classified as No language but C++ was expected
Field.h is classified as Objective-C but C++ was expected
gdsdbreader.h is classified as Objective-C but C++ was expected
key.h is classified as C but C++ was expected
libcanister.h is classified as Objective-C but C++ was expected
Math.inl is classified as No language but C++ was expected
metrics.h is classified as Objective-C but C++ was expected
protocol-buffer.pb.cc is classified as No language but C++ was expected
protocol-buffer.pb.h is classified as C but C++ was expected
qscicommand.h is classified as Objective-C but C++ was expected
qsciprinter.h is classified as Objective-C but C++ was expected
rpc.h is classified as Objective-C but C++ was expected
scanner.cc is classified as No language but C++ was expected
scanner.h is classified as C but C++ was expected
Types.h is classified as Objective-C but C++ was expected
utils.h is classified as C but C++ was expected
v8.cc is classified as No language but C++ was expected
v8.h is classified as C but C++ was expected

Changes between 19c46da7...862bc3c33
benchmark/samples/C++
0.21% change to No language (count files)
```

Overall I'm reasonably happy with this implementation but there are some issues right now I could do with some help on:

1. I'm having to do [something funky](https://github.com/github/linguist/commit/19c46da714de1140d13e8b7e931299782c8c05b4) in `Rakefile` to try and force a reload of Linguist. This doesn't seem to be working properly as if I run `bundle exec rake benchmark:run index` for the respective commits (19c46da7 & 862bc3c33) then there should be fairly major changes due to [this very destructive commit](https://github.com/github/linguist/commit/862bc3c337f1b2b5eb5be7c8a83f95084663a28f).
2. Testing is a pain with all the `git checkout` stuff. It feels like there should be a better way.

Anyway, I'm off on me holidays for a couple of weeks but will pick this up then.

/ cc @bkeepers