Not at all! Many people make vanilla, coffee, cajeta, or horchata raspados. Use whatever flavors you think will taste the best.