Fix #3647 [backport to 1.4.x]

This fixes #3647, for me on the tkagg backend.
What still is strange is that the bug does not occur for e.g. cairo backend.
Actually I do not understand why this is inelegant, compared to tracking the clip values in the dictionary all the time. _bbox is not the Rectangle itself or something but just a dict.