I am taking a break from pounding my head against the wall to bring you this comment.  There is an order of precedence defined in the build method that is not documented.  
1. display
2. control
3. widget
4. current display

It probably makes sense to modify this in one of the following ways :
1. throw an UnsupportedOperationException on subsequent calls to set such that only one set method can be called.
2. Simplify the interface such that it only takes a display and let the user call control.getDisplay() to set
3. Get rid of the builder altogether as Ben suggested 

Aside, I had to disconnect from VPN to place this comment because github is unusable when connected.