Do less work in setSize

Issue #1682