No, you don't put you'll be missing out on a lot of additional flavour. Instead of grapes, you could substitute them for strawberries or peaches.