If you're toasting the nuts before using them, be careful not to over-toast them. This can lead to a bitter taste. Toast them just until they're golden brown. Then let them cool completely before adding them. The quality of the olive oil also impacts the taste. If you use low-quality or rancid olive oil, it can result in a bitter flavor. Use high-quality extra virgin olive oil to get the best taste.