You can, but the soft brown sugar helps both flavour and texture, so I don't advise it.