Fix keyboard readding keys

If you manually remove the key from the buffer and the key is still held,
it used to be readded to the buffer. This fixes it.