You may still eat a cauliflower that has brown spots. Use a sharp knife to simply trim the brown or yellow spotting and use the rest.