WIP: step 19 - realised that the unit test used to replicate the problem seen at the client site may well be just trivially causing an out-of-memory through the sheer number of test variables. While this is itself a useful case to consider, it isn't the same problem as seen at the client site - the latter actually sees full test vectors, albeit very slowly, whereas the test here is simply bombing out before *any* partial test vectors are presented for merging. Have cut down the test to see if I can observe a difference in running time before and after optimising via shared path prefix.
WIP step 1: introduce replacement for 'Map' - 'MapWithRunLengths'. Currently just an minimal API with no implementation.
WIP step 2 - incomplete (and incorrect) implementation of 'MapWithRunLengths'. Has some stuff worth salvaging, so I'll make this checkpoint commit.
Correct spelling typo.
WIP step 3 - starting to pass some tests, but some of the tests are either spinning in infinite loops or are taking a *very* long time to run. The implementation really could do with sprucing up - but in it goes as a checkpoint.
WIP step 4 - some fixes resulting from self-review of the implementation of 'MapWithRunLengths'. Still have at least one outstanding bug and unimplemented functionality on top of this.
WIP step 5 - fix for incorrect bug-fix in previous commit (hmm) plus some more unimplemented functionality. Still got major test breakages.
WIP step 6 - implement more missing pieces, still have big test breakage which is confirmed in some tests that now break non-trivially with the extra implementation.
WIP step 7 - more bug fixes, passing the Node unit tests now, but still have breakage for the all-up tests.
WIP step 8 - add in some temporary postcondition checking - not very efficient and messy code duplication, hence the temporary nature. Postconditions are not violated despite the ongoing test failures.
WIP step 9 - more postcondition checking. Still got breakage - but now think it's from elsewhere!
WIP step 10 - bugfix for change made at the base of this branch - I had assumed a collection was ordered when it wasn't. This seems to have solved one bug and unearthed another - so the tests now break in a different way. Investigating...
Merge in bugfixes for problems originally spotted on this branch but present in the parent branch (and thus fixed on a separate bug-fix branch sharing the same parent as this).
WIP step 11 - amended postcondition for'MapWithRunLengths.Add' to take into consideration the no-operation case where an existing key-value pair is added into the same map repeatedly.
Turned up the dials on test 'StressCopiousUseOfInterleavesMotivatedByBuildingLogicalExpressionTestCases'. Causes an OOM at around 1.5G of memory usage!
Upmerge from 'main' - bring in the bugfix for filling out partial test vectors.
Merge in work from 'mapWithRunLengthsExperiment' to see if it makes any difference to the scaling OOM: well - it does, it takes an extra 1.5 minutes before it reaches OOM on a 2.133M virtual machine. Maybe this is progress, the memory usage profile seems to have changed...
Turned down one of the test dials to get a more useful test - this time we actually get to merge some partial test vectors, plus added in building with tail calls enabled that is required to get the fusion algorithm in 'MapWithRunLengths' to run on the resulting long partial test vectors.
Exploratory hacks to cut down on memory usage explosion when presented with a huge number of leaf test variable nodes. Encouraging results, but the stressed interleave test shows a rather obvious skew when inspecting the generated test cases. Better to revisit this using a randomised cross product generator for both test variable index combinations and test variables - this would also allow the removal of the old randomisation hacks to encourage the early production of full test case vectors.
Introduced test intended for 'DecorrelatedCrossProduct', although currently applied to the existing cross product - and therefore failing nicely. Also corrected some poor nomenclature in the existing cross product implementation.