Move drawing logic to drawers

Close #1106.
Do we want to mirror d3's logic here?