In Watch Later, click the 3 dots menu to remove a video. This only removes the bookmark link, not the video itself.