Fix #3647 [backport to 1.4.x]

This fixes #3647, for me on the tkagg backend.
What still is strange is that the bug does not occur for e.g. cairo backend.
This should be done when the bounding box is created and when the various `self.*clip*` properties are done, not on every draw.  This _does_ fix the problem, but is rather inelegant.