Only remove directory if it's empty