Make sure the directory is empty even if weird inputs are given, fixes #1683