Descend into the actual directory, just in case the homedir is a symlink