Not a problem, you definitely can enjoy this bread without olives, or replace them with walnuts (if you can have them) or dried tomatoes, for instance!