Tzatziki should taste refreshing, herby, zesty and indeed slightly sour, as it is made of sour ingredients like Greek yogurt and lemon juice. If you want, you can add ½-1 teaspoon of sugar to balance it out. If it tastes bitter, then the cucumbers and/or lemon may be past their best.