opened
jameshensman
username_0
I added attributes to the tests to mark the ones that are slow. You can now skip them with

`nosetests -A "speed!='slow'" `

note that all the tests still run on Travis/Codeship. If anyone has any good ideas for speeding up the turnaround time there, I'd be happy to hear them.
Test speeds
issue
created
javdrher
username_1
The GPflowOpt tests are getting lengthy on travis so I have been looking at this. I'm currently adding caching of dependencies, should save about 30s

You could add slow/!= slow as envs and split the build. Those two can then be parallelized as described here: https://blog.travis-ci.com/2012-11-28-speeding-up-your-tests-by-parallelizing-them/
Both builds will still run, but the fast build will provide feedback faster.
comment
created
jameshensman
username_0
So, in investigating the tests, I discovered that the notebook tests were not actually running. This is because the path to the notebook directory was hard coded relative to the working directory, which changes on the Travis/Codeship server. 

I fixed the relative path issue, so now the notebooks get tested as part of the CI. 

I has to blacklist the regression notebook, because the MCMC cell times out on Codeship (the free server is very underpowered). But this PR means that more is being tested, not less.
comment
created
jameshensman
username_0
Thanks @username_1 , would be great if you could put those changes in another PR. 

@username_2 , could you give the PR a quick check to make sure it makes sense? Thanks.
comment
created
javdrher
username_1
@username_0 I'll give it a try for GPflow. I'm not very happy about the speedup achieved for GPflowOpt. Most of the time seems to be lost while the build is waiting to be started.
comment
created
alexisboukouvalas
username_2
I am away this week but can happily take a look this weekend
comment
created
jameshensman
username_0
@username_2 marvellous, thank you.
comment
created
alexisboukouvalas
username_2
As mentioned in #437, a big speedup could be achieved if the notebook tests was parallelised. Conceptually this is simple because the each notebook test is independent. So @username_1 it seems you already have a good implementation for this. Could you create a pull request or even merge with this one?

I would prefer to remove the 'blacklist' feature if possible.
Is the problem with the GPLVM notebook, import pods? Can we add this to tests_require or I could change the script to use local csv file. I have found at least on bug using the GPLVM notebook so I think it would be worth keeping in (#377).
Could we add a 'very slow' attribute that would run all the notebooks?
comment
created
javdrher
username_1
The conclusion of #437 is that hardly any speed is gained by running tests in parallel on Travis. I believe a build has access to something between 1 and 2 cores. Because calculations occur in Tensorflow this capacity is more or less consumed by a single test. 

At least, thats for Travis. Locally you can gain quite some speed. If interested I can apply the test parameterization to run a notebook one test each.
comment
created
alexisboukouvalas
username_2
thanks @username_1 
I was thinking about the local case to enable running all tests quickly including the notebook tests.
If you upload your local tests, I can give it a go and see the performance difference on a couple of different machines/configurations.
Just making the notebook tests faster should not significantly increase test complexity right?
comment
created
jameshensman
username_0
@username_2 , if this ticket is all correct, could we merge? Improving the notebook tests further seems out of the scope of this ticket. This does provide a means to run most tests quickly, and notebooks will still be tested on travis/codeship. Thanks.
comment
created
alexisboukouvalas
username_2
Of course these fixes improve the current tests
comment