Extra virgin olive oil (EVOO) has a very strong flavour that will overpower the delicate vanilla flavour. If you want to substitute with olive oil, choose a light tasting olive oil instead.