You certainly can. Whilst I love including ginger for a subtle kick of extra flavour, feel free to omit it if you don't like the taste.