Yes, you can omit the saffron entirely if you prefer, or substitute 1/4 teaspoon ground nutmeg for the saffron. The buns will still have a yellow hue, thanks to the turmeric, but will not be quite as golden in color without the saffron.