You do not need Fleece to level Weaver, but you do need it to finish the Weaver class quests unless you want to buy the items that you need instead.