All standard gear above ilvl 50 is automatically scaled down to ilvl 50 while inside the Mage Tower scenarios. This means that your gear, trinkets and stats do matter, but all your gear is scaled down to a maximum of item level 50. As long as a piece of gear is over item level 50, that is the highest item level that can be achieved on a piece of gear inside the Mage Tower. Because the Mage Tower is scaled to Item Level 50, this removes outgearing the content in the normal fashion - item level. However, you can still make the Mage Tower significantly easier by using a Timewalking set - equipping pieces of gear that are overbudgeted for their slot when scaled to a lower item level, which you can start farming now!