If the work is uncomplicated and both parties agree on the details, then there wouldn't be an issue, but experience tells us that generally this isn't the case, and it's short-sighted to start work without an agreement in place.
The Party Wall Act may seem a bit of a drag, but it offers legal protection to both sides. The building owner would give up their rights to work and entry, as well as certain expenses, and the adjoining owner would lose:
their right to have to work fully defined;
their right to serve a counter-notice and maintain their easements;
as their right to security for expenses.
Both sides would also lose the machinery of settling the dispute with a professional surveyor too, so the benefits of having a party wall agreement are clear.
At Lapworth Architects, as well as providing you with the relevant drawings for your application, we can help through this admittedly complex procedure to provide you with a Party Wall Agreement