Moving some time handling to pygame, fixing pygame shutdown