Capers taste similar to olives but have lemony undertones and a bit more tang. While olives can be eaten on their own, capers are usually better when combined with other ingredients to balance out their robust flavor.