Maintain an internal model of the address pins, and only use I2C when actually necessary, to speed things up.
Make bare minimum use of I2C communication with port expanders by making the firmware smarter, to speed things up.