The best explanation for why you can't go back in time is still the causality objection. You could kill your grandfather and thus never exist, etc. Unfortunately this is still a philosophical objection rather than one rooted in physics. The equations in physics seem to work the same with time going forward or backward. Because of entropy.
Ice cubes melt in liquid water.
A glass of cold water will never spontaneously form an ice cube. The Second Law of Thermodynamics says these things happen in only one direction. You can't fight it. You can't reverse time because you can't make things become more orderly rather than less orderly. The Earth goes around the Sun.