[ci] Nix pygame req of parent and attempt cache