Arithmetic went on perfectly well for 2000 years without numbers being conceptualized as sets. But if you use that metaphor, then interesting mathematics results. There is a third less well-known metaphor for numbers, that Numbers Are Values of Strategies in combinatorial game theory. These metaphors for numbers are part of the mathematics, and you make a choice each time depending on the kind of mathematics you want to be doing. It's a perfectly sensible idea.
Conceptual metaphors are cross-domain mappings that preserve inferential structure.
Mathematical metaphors are what provide the links across different branches of mathematics. One of our most interesting results concerns the conceptualization of infinity. What mathematicians have done is to provide elaborate carefully devised special cases of this basic metaphorical idea. What our results appear to disprove is what we call the Romance of Mathematics, the idea that mathematics exists independently of beings with bodies and brains and that mathematics structures the universe independently of any embodied beings to create the mathematics. This does not, of course, result in the idea that mathematics is an arbitrary product of culture as some postmodern theorists would have it.