Revert "Add defconfig for caf builds"

This reverts commit 16d98ae174dae594e2236c07f720ae5f445bcd9e.