pybrain/pybrain — pull request #96 by bjerva (2013-06-09 08:13:44+00:00)
Title: Using splitWithProportion on large data sets is time consuming

Using splitWithProportion in e.g. ClassificationDataSet took forever. This seems to have been caused by checking "if *element* in *list*", which is far from optimal when the data set has over a million samples in it.

I changed the random sample list which is used so that it instead is a set. Granted, this does increase the memory footprint, but I think this is preferable for most people.

The speed-up using a set appears to be quite substantial. The old method using a list did not complete in several hours, whereas the new method only used a few seconds.