Tests: Remove obsolete speed directory

Closes gh-1551