Tests: Remove obsolete speed directory

I think there may still be a stray reference to `./speed` in the build process so it can be excluded.