It lies in the respective subject matters of mathematics and physics. In the latter case we are interested in real world phenomena. In the former we address ethereal abstractions which are not of this world. The brute methodological disparity motivates a classification of propositions as either a priori or a posteriori , depending on whether or not observations are thought essential for determining truth. The explanation lies in a difference of content in the proposition.
Mathematical propositions say nothing about the real world.
Scientific propositions by contrast are called synthetic. The say something substantive about the world. One further step may be taken along this road. If the truth of a proposition can be determined without observing the world, then our grounds for belief in the proposition will hold good in any world. Prior to observation all worlds are, for all we know, identical.