The bananas you used for the pudding mixture must have been already overripe. Consider using bananas that are still fresh without any hints of bruises on their skin, as overripe bananas tend to give off the sour notes in the flavor.