Remove empty build and spec directories ( on second thought I'm an idiot )