Remove speed validation for kernels where it's done in the spi core