Unless your entire army has been slaughtered and the enemy has multiple Battering Rams smashing down the front door, you can count on holding your position. The Castle's destructive firepower forces the enemy to consider attacking elsewhere and from a different direction. Furthermore, you can Garrison the Castle, which makes it virtually impenetrable. Any player who hasn't taken the offensive by the Imperial Age is probably in trouble. The Imperial Age completes a balance shift, giving the attack a distinct advantage.
Units become stronger than structures, far more powerful than in earlier ages.
These units outrange all defensive structures, rendering them vulnerable to their barrages. Military units offer the only means of countering these offensive dynamos. Relying on defensive structures during the Imperial Age will doom you to failure. It is extremely difficult to wage war during the Dark Ages. It is unlikely that the units you could afford to create this early in the game could withstand the firepower of your opponent's Town Center, let alone any defenses that might have been erected.