Kill off some casting in the FifoPlayer.

Also makes more sense conceptually, since it's not like we'll ever have negative frames.