chapel-lang/chapel — pull request #295 by bradcray (2014-09-03 22:27:03+00:00)
Title: Add best practices instructions for creating performance tests

The main goal here was to create a performance test by updating the
bestPractices/TestSystem.txt to go through the steps required to
create a minimal interesting performance test with graph. While
editing that file, I also did some wordsmithing to the previous
version and added information about .bad files for .futures.

In documenting this, I ended up creating a performance test to make
sure I had everything right, and then realized that putting this into
the test/ system for other developers to see (rather than only having
it be in the .txt file) would be valuable, so I added
Samples/Performance/ as a place to capture it. I created Chapel and C
versions since that is a common and interesting idiom and I made a
.graph file for them, even though they plot completely bogus (and
static data). I added it to GRAPHFILES so that we could make sure it
didn't break over time after nobody objected on the perf mailing list
or Cray chatroom.

Sung pointed out that the Samples/ directory has traditionally been
treated specially in start_test (it skipped over it, unless you asked
it not to), and as a result, Samples/ was very stale and not helpful.
So I blew away the existing Samples/ directory, and added the
(trivial) correctness test sample that was in the TestSystem.txt file
in a Correctness/ subdirectory. Over time, I think it would be great
to document other aspects of the test system in TestSystem.txt and
illustrate them in the Samples/ directory but didn't want to get
pulled into that today.

So then I stopped having start_test treat Samples/ directory and
updated README.dirs in test/ to reflect that it's not so special
anymore.