Sure they may advance a bit sooner than you because they didn't have peons on building walls, but they will waste time getting through your walls. Take your large group now and attack him. Do not build a cat and start taking his wall out. All you are doing is letting him know you are there which makes him pump troops out or bring his troops back that are out in the field to defend. You want to surprise him.
Surprise is much more effective.
You should be ready to start going iron now. Don't keep cranking chariots out if your walls are still safe. I mean if you have hurt him already he won't advance as fast you anyways, so there is no need to keep cranking out wimpy bronze units. Now send all your bronze units to attack his base while you waiting for the advancement for iron to finish. If he has a large army waiting in his base fight the army, BUT break 3 chariot archers from the BACK OF THE BACK .