If the Y chromosome changes that part of the brain, those breasts will not develop in men. Breasts are how males and females are the same but there are many other places where they are different. The reproductive organs are one obvious place. But there are others, the knee joint is different in the two genders, males have Adam's apples that females do not. The other area that is different is the brain.
Every animal has a sexual orientation.
If they did not they could just as easily go around humping trees to get physical stimulation and then DNA would never get reproduced. It is the gene, the unit of heredity. Richard Dawkins has presented this idea best in his book the Selfish Gene. His argument is that all other animals, and we are machines created by our genes. The same is true with sexual orientation.