Technically, you don't need to remove the seeds, but for the sake of creaminess, I do remove them for kumquat pie.