In time, many of the Fujiwara were replaced, mostly by members of the rising Minamoto family