Honestly, it's not something that I personally put a lot of stock in, and I often don't bother doing so throughout my playthrough. The more convenient option is to simply go to Pierres and pay a little over the odds for the ingredients you need. I would only concede that those who really love cooking in this game would benefit from this building, as this will save you a bundle in the long run. However, this is only really relevant in the early to mid-game, and I often don't bother making it a priority to make cloth. Then by the time I have this resource naturally, spending 100G on some ingredients now and then is chump change. In short, it's not a big deal if you don't build yourself a mill.