First, the basic elements on which our mathematical constructions are based, such as numbers, sets, space, and so on, have been rooted in the architecture of our brains by a long evolutionary process. At the scale we live in, number is essential because we live in a world made of movable, denumerable objects. Things might have been very different if we lived in a purely fluid world, or at an atomic scale and hence I concur with a few other mathematicians such as Henri Poincare, Max Delbruck, or Reuben Hersh in thinking that other life forms could have had mathematics very different from our own. But then they have been selected for their usefulness in solving real world problems, for instance in physics. Hence, many of our current mathematical tools are well adapted to the outside world, precisely because they were selected as a function of this fit.
Many mathematicians are Platonists.
They think that the Universe is made of mathematical stuff, and that the job of mathematicians is merely to discover it. I strongly deny this point of view. I agree with Gardner, and against many social constructivists, that mathematical constructions transcend specific human cultures. Furthermore, I am in no way denying that the external world provides a lot of structure, which gets incorporated into our mathematics. We develop mathematical models of the world, but these are only models, and they are never fully adequate.