TeamAmalgam/test-models — pull request #2 by mhyee (2014-01-12 22:49:48+00:00)
Title: New test models

Fixes #1

The idea here is to make our test models build up from each other. So knapsack_20_metrics_4 is basically knapsack_10_metrics_2 with ten more items, and two more metrics each. The first 10 items in both of those problems are the same. This way, we can more easily compare performance across problems (knapsack_10_metrics_2 is strictly a subset of knapsack_20_metrics_4, so it should take less time to solve).

`values.yaml` was created by running `generate_values.rb`. It's simply a YAML file with a lot of random numbers that we can reuse. There are 100 sets, each with 10 metric values.

All the knapsack, rooks, and queens problems were regenerated. There's nothing fancy with the knapsack problems.

The chessboard problems are a bit more complicated. I wanted the smaller boards to be "contained" within larger boards (with rotation allowed). To accomplish this, I "fill" the board in spiral order. For example, consider the 3x3 and 4x4 boards below:

8 7 6
1 0 5
2 3 4

15 14 13 12
4 3 2 11
5 0 1 10
6 7 8 9

So square 0 on both boards has the same metric values.

I did some quick diffs, comparing different-sized problems to each other, to ensure they were pretty much the same. I also ran a few of these through the Alloy Analyzer (after removing the multi-objective stuff) to ensure that they compiled, and I compared the number of variables and clauses to the old test models, to confirm that they were the same (since the only difference should be the metric values). I also ran a few problems through Moolloy, and verified rooks_3_metrics_3 and queens_4_metrics_2 by hand.

The final test would be to run all of these problems through Moolloy, but I have no idea how long that would take.

Note that this pull request also includes queens_3* problems, which currently fails due to a bug in Moolloy ( TeamAmalgam/kodkod#9 )

Note also that we'll have to clean out the dashboard.

Reviewers: @AtulanZaman @cpkleynhans @joseph2625