The Role Of Urf
The cultural norms decide it. In every culture certain dresses are considered gender specific, while there are others which are considered allowed for both to wear them.