Postpone deleting invalid frames until we are permitted to do work in the GL context