Yes the scorer API still has a few glitches (see #2588).

>  I just can feel that there are a bunch of issue with the pipeline API, that we need all to write them in a design discussion document, accumulate usecases and someone brave needs to figure out how to move to the next step.

This is clearly important but we also have to accept that we can't always get the perfect design on our first attempt. Some of our best decisions were incremental and emerged from previous mistakes (e.g., there used to be separate classes for dense and sparse estimators). I think we should merge important API changes to master at the *beginning* of release cycles. This way, core developers and early adopters get to play with the API and we can catch potential mistakes.