Fix gpio_matrix cpu hog while using the keyboard