<PAGE>
                                                                    Exhibit 10.4

                                 AMENDMENT NO. 3

                                       TO

                             SHAREHOLDERS AGREEMENT

          AMENDMENT NO. 3, dated as of June 20, 2005 (the "Amendment No. 3"), to
the SHAREHOLDERS AGREEMENT, dated as of November 21, 2001, as amended from time
to time, among the Shareholders named therein and that may become parties hereto
from time to time hereafter (the "Shareholders Agreement"), and to which Allied
World Assurance Holdings, Ltd, a limited liability company organized under the
laws of Bermuda (together with any successor thereto, the "Company"), has been
made a party.

                                   WITNESSETH

          WHEREAS, in order to carry out the full intent of Amendment No. 2 to
the Shareholders Agreement, Section 2.3(b) of the Shareholders Agreement must be
deleted;

          WHEREAS, the Company and each of the Founders, Securitas and the
Non-Founder Shareholders holding at least the requisite number of outstanding
Common Stock desire to amend the Shareholders Agreement, pursuant to Section 7.7
thereof, as set forth in this Amendment No. 3; and

          WHEREAS, all capitalized terms used but not defined in this Amendment
No. 3 shall have the meanings set forth in Annex A to the Shareholders
Agreement;

          NOW, THEREFORE, the Shareholders and the Company agree as follows:

     1.   Section 2.3(b) is hereby deleted in its entirety and the following
          sentence shall be inserted in lieu thereof:

               "Intentionally omitted."

     2.   The Shareholders Agreement, except as amended by this Amendment No. 3,
          shall remain in full force and effect in accordance with its terms.
          All references to the "Agreement" contained in the Shareholders
          Agreement shall be references to the Shareholders Agreement, as
          amended.

     3.   This Amendment No. 3 shall be governed by, and construed in accordance
          with, the laws of the State of New York.

<PAGE>

     4.   This Amendment No. 3 may be executed in any number of counterparts,
          each of which shall be deemed to be an original and all of which
          together shall be deemed to be one and the same instrument.

          IN WITNESS WHEREOF, the parties hereto have duly executed this
Amendment No. 3 to the Shareholders Agreement as of the day and year first above
written.

                                        ALLIED WORLD ASSURANCE HOLDINGS, LTD


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        AMERICAN INTERNATIONAL GROUP, INC.


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        THE CHUBB CORPORATION


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                       -2-

<PAGE>

                                        GS CAPITAL PARTNERS 2000, L.P.

                                        By: GS Advisors 2000, L.L.C.
                                            Its General Partner


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        GS CAPITAL PARTNERS 2000 OFFSHORE, L.P.

                                        By: GS Advisors 2000, L.L.C.
                                            Its General Partner


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        GS CAPITAL PARTNERS 2000 GmbH& CO.
                                        BETEILIGUNGS KG

                                        By: Goldman Sachs Management GP GmbH
                                            Its General Partner


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        GS CAPITAL PARTNERS 2000 EMPLOYEE FUND,
                                        L.P.

                                        By: GS Employee Funds 2000 GP, L.L.C.
                                            Its General Partner


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                      -3-

<PAGE>

                                        STONE STREET FUND 2000, L.P.

                                        By: Stone Street 2000, L.L.C.
                                            Its General Partner


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        BRIDGE STREET SPECIAL OPPORTUNITIES FUND
                                        2000, L.P.

                                        By: Bridge Street Special Opportunities
                                            2000, L.L.C.
                                            Its General Partner


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        SECURITAS ALLIED HOLDINGS, LTD


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                        GOLDMAN, SACHS & CO., on behalf of each
                                        of the Non-Founder Shareholders listed
                                        on the attached Annex A, pursuant to the
                                        respective power of attorney granted by
                                        each such Non-Founder Shareholder


                                        By:
                                            ------------------------------------
                                        Name:
                                              ----------------------------------
                                        Title:
                                               ---------------------------------


                                       -4-