Absolutely, you can skip the lavender in the ganache. The chocolate ganache will still complement the cake beautifully.