Yoritomo then turned his attention to the elimination of the powerful Fujiwara family, which sheltered his rebellious brother Yoshitsune