Olive oil has a distinctive taste that adds richness and boosts the flavor of the lemon in this cake. Olive oil can lose some of its flavors when baked. Use a lighter type of olive oil than extra-virgin olive oil if you want a less intense olive oil flavor.