You always have the option to ignore references, so that they are not made visible on your profile. You can ignore references via the reference page, by clicking on the ignore button next to the reference. Even if you accept a reference to be shown on your profile, you can always choose to ignore it later.