You can but olive oil really shines here and works well with the other flavours. This is, after all, an olive oil cake!