[brcm63xx] Move SPI driver out of patch.