No, your neighbour cannot refuse to Party Wall Agreement. Your neighbour must initially acknowledge the Party Wall Notice by either consenting of dissenting to the property works.
If your neighbour does not provide consent or dissent to the Party Wall Notice, an automatic ‘dissenting’ will occur Notice. Whilst your neighbour cannot stop the works from being undertaken, a dissent to the Notice ‘triggers’ a dispute under the Act.
An Agreed Surveyor or Surveyors will need to be appointed in order to resolve this dispute. The Surveyors will agree on the wording of the Party Wall Agreement which sets out the timing and manner of the proposed building works.
Once the Party Wall Agreement has been served upon the building and adjoining owners works can commencement, subject to the expiry of the relevant Notice period.