Remove key listeners from board on init