Remove directories more safely

If you don’t have build/ or cache/ and exit without running anything,
an error was thrown before because we tried to remove non-existent
directories.