yui/yui3 — pull request #1029 by derek (2013-07-25 02:54:04+00:00)
Title: Perf Test Refresh

# Perf Test Refresh

A project I've been working on recently has been to standardize and automate YUI's performance testing. While the tools to do this are not yet released (soon), there are some modifications that will need to be made to YUI's current performance tests. Included in this pull request are proposed modifications and alternative ideas.

First...

## Problems with the current tests
Currently the source tree has various HTML+JS combinations that comprise performance tests ([an example](https://github.com/yui/yui3/tree/master/src/app/tests/performance)), mostly using [benchmark.js](http://benchmarkjs.com/). While this has worked fine for manual testing against your current tree, it's proven to be a little challenging to test in more automated and dynamic ways.

To summarize a few of the issues with current tests:

* Requires lots of boilerplate, which increases the overhead of writing new tests, thus reducing the test coverage.
* Test files are difficult to parse. What is boilerplate and what is test code?
* Written specifically for Benchmark.js, so they aren't portable

The parsing aspect of it is very important since we're reconstructing performance tests with different seeds for cross-version testing. Also, since performance metrics can guide development priorities, we need to continually experiment with different testing strategies and tools to ensure the results are as reliable as possible. Unit testing is simple pass/fail, but performance testing asks "How good is your code?" The answer to that question is continually evolving.

## Solution

To address this, we need a slighty new way of writing tests. Since the compiler takes care of writing all the boilerplate for you, the only thing we're interested in are the basic ingredients for a performance test: (1) suite metadata and (2) test functions.

There are two directions we can go here, imperative or declarative. The files contained in this pull request represent the declarative approach.

### Option A

I started with this approach as it is just a simple reflection of what the objects are parsed into inside the test compiler. The general idea is you start with an object, and fill it with a few basic properties all perf tests have.

For example:

// config.js
{
title: 'My test suite',
tests: [
{
title: 'My test',
fn: function () {
// do something
}
}
]
}

There are some additional properties you'll find included in this PR (check em out), but that is the most basic example and only contains the required fields.

### Option B

The imperative approach would involve wrapping parts of these tests in an API. Something like:

// config.js
var suite = new PerfTest('My test suite');

suite.addTest({
title: 'My test',
fn: function () {
// do something
}
});

So, it would probably look similar to the [Benchmark.js API](http://benchmarkjs.com/docs). Likely just a subset of it. In this world, a testing tool would have a simple API to collect each component of a suite, then assemble it into a format ultimately similar to the declarative approach.

For example, the collection class would look something like:

function PerfTest (title) {
this.suite = {
title: title,
tests: []
};
}

PerfTest.prototype.addTest = function (test) {
this.suite.tests.push(test);
};

Simple, but it will get a little more complex as the instance will need to support declaring `setup`/`teardown`, prep HTML, etc.

## Preference?

We can easily go either direction. For a true apples to apples comparison, here is the `node-create` test written in [Option A](https://github.com/derek/yui3/blob/new-perf/src/node/tests/performance/node-create.js), [Option B](https://gist.github.com/derek/1b64542f9b679b454b54), and the [old format](https://github.com/yui/yui3/blob/v3.11.0/src/node/tests/perf/node-create-benchmark.js).

From the compiler's perspective, it all ends up looking like **Option A** before test construction, so the question just comes down to how do developers want to write tests?

My personal preference is to go with **Option A**. Why?

* It is a simple, unopinionated approach
* When perf testing, we spend most of our time tweaking library code, not writing tests, so elegance takes a backseat to simple and straight-forward.
* It doesn't blur the lines between implementation vs configuration
* Is dead-simple to parse, so it can be turned into anything
* Doesn't require learning a new API, just a few field names
* One suite per file (forced) is nice, the filename becomes the slug

But if the preference is to go with **Option B**, I can write a simple parser to crawl through these files and reconstruct the suites with the new TBD API.

Discuss...