Use rmdir instead of rm -rf

This way make clean runs more... cleanly

* I had a lot of warnings that bin/ is not empty. This way they are avoided.