Fix #3647 [backport to 1.4.x]

This fixes #3647, for me on the tkagg backend.
What still is strange is that the bug does not occur for e.g. cairo backend.
@tacaswell: I'd appreciate, if you comment on that.