Optimize ImplCheckSizeOfSwappedInGraphics() a bit

Store used size as a member so we don't need to recalculate it
all the time and no need to use a size map.

Change-Id: I1f929c5d3a56f545cef123bda087ecaf8ca0be4a