Include all released buttons in the latch.

Not just newly released ones.  This does appear to be how the PSP fills
this buffer - it's normally 0xffffffff.