<PAGE>
 
                                                                    EXHIBIT 10.4

                    IRREVOCABLE PROXY AND VOTING AGREEMENT

     I, Gail E. Granton, a shareholder of Pacific Gateway Exchange, Inc., a 
Delaware corporation (the "Corporation"), individually and as the trustee of the
Granton Foundation (the "Foundation") hereby appoint Howard A. Neckowitz to be
my proxy agent and the proxy agent of the Foundation and to vote all of my
shares in the Corporation and all shares in the Corporation held by the
Foundation (the "Shares") with respect to all matters submitted to the
stockholders at all meetings of the stockholders, or any adjournments thereof,
and in all consents to any actions taken without a meeting. This appointment
shall commence immediately and end on January 1, 1999. During said period,
Howard A. Neckowitz shall have all of the power that I or the Foundation would
possess with respect to the voting of my shares and all of the Foundation's
shares and granting my consent and the Foundation's consent. I hereby ratify and
confirm all acts that the proxy pursuant to this Irrevocable Proxy and Voting
Agreement (the "Proxy") shall do or cause to be done by virtue of and within the
limitations set forth in this Proxy.

     If, during the term of this proxy, Howard A. Neckowitz becomes unable to 
perform his duties as an officer of Pacific Gateway Exchange, Inc. due to his 
voluntary resignation, termination for cause, long-term disability, physical or 
mental incapacity, or death, then this Proxy shall be null and void. Except as 
set forth in this paragraph, this Proxy shall be irrevocable.

     I agree that any transfer of the Shares by me or the Foundation prior to 
January 1, 1999 other than pursuant to a registered underwritten offering under 
the Securities Act of 1933, as amended, (the "Act") or pursuant to Rule 144 
under the Act shall be subject to this Proxy. It shall be a condition of such 
transfer that the transferee appoint Howard A. Neckowitz as such stockholder's 
proxy on terms identical to this Proxy for the term ending upon termination of 
this Proxy.

     I agree that the Shares shall bear the following legend until this Proxy 
terminates as set forth above:

          The shares represented by this certificate are subject to a
          Irrevocable Proxy and Voting Agreement dated as of April 30, 1998, a
          copy of which is on file with the Secretary of the Corporation. Any
          purchaser of such shares prior to January 1, 1999, other than pursuant
          to a registered underwritten offering under the Securities Act of
          1933, as amended, (the "Act") or pursuant to Rule 144 under the Act
          shall be bound by such agreement.

     IN WITNESS WHEREOF, I have executed this Proxy effective as of the 30th day
of April 1998.

                                   /s/ Gail Granton
                                   ----------------
                                   Gail E. Granton

                                   The Granton Foundation


                                   By: /s/ Gail Granton
                                       ----------------
                                       Gail E. Granton
                                       Trustee