Make builder listeners protected, set them in the constructor

Builder should just create the instance with the builder, setting these listeners should be done in the constructor. Made the listeners protected to match the other fields, and make them available to subclasses.