Handle cwd refcount a bit more properly