Nope, don't worry about removing them. They actually continue to give the vanilla a wonderful flavor.  I actually scraped the seeds from the canning jar and made sure that some got into each individual bottle that I gave away.