Before applying new caulk you usually do need to remove the old caulk. New caulk doesn't stick to old caulk so normally if the old one is not removed it won't seal properly.