travis-ci/travis-ci — Issue #352: [Feature Request] Continuous Performance Testing in travis-ci
Author: dscape
Created: 2011-12-12 08:37:40+00:00
State: open

In some modules performance is the key aspect to evaluate. It would be great if `travis-ci` allowed users to performance test their applications.

As an example I'm building a [spell checker][spell] module for node. There's really not much to testing the functionality and interfaces are unlikely to ever change. The real important factor to test in a spell checker is speed. Namely how it evolves from version to version, and making sure that a pull request does not statistically degrade performance of the library, or proving that it actually improves performance in a statistically significant way.

This is how I see the workflow:

* Either we choose a performance branch we test, or we test everytime a new tag (version) exists
* When that is triggered we run tests. We need to make sure they compare to previous versions so we actually might need to run tests for all version everytime. Or maybe just the current version and previous one is enough.
* If there is a statistically significant difference in performance in one (or more) methods than use should receive an email saying performance is worse. If the performance has actually increased user should be sent an email saying so too.
* In the webpage we could have a up arrow, down arrow, equal sign to show the progression of the performance for this module

I know this is super hard work and I really appreciate all the awesomeness that has gone into travis already. I just think this would be super useful to help the open-source community make better modules that work blazing fast.

/cc @3rd-eden @mmalecki @isaacs

[spell]: https://github.com/dscape/spell