<PAGE>

                                                                    Exhibit 10.4

                        PARTIAL PAYOFF, AGENT REPLACEMENT
                          AND ASSIGNMENT AND ASSUMPTION
                                    AGREEMENT

          PARTIAL PAYOFF, AGENT REPLACEMENT AND ASSIGNMENT AND ASSUMPTION
AGREEMENT (this "Agreement") dated as of May 17, 2002 among TELETOUCH
                 ---------
COMMUNICATIONS, INC., a corporation duly organized and validly existing under
the laws of the State of Delaware (the "Company"), each of the lenders that is a
                                        -------
signatory hereto under the caption "LENDERS" on the signature pages hereto
(individually a "Lender" and collectively the "Lenders"), JPMORGAN CHASE BANK
                 ------                        -------
(successor to The Chase Manhattan Bank), as administrative agent for the Lenders
("Chase" or the "Existing Administrative Agent"), and ING PRIME RATE TRUST
  -----          -----------------------------
(formerly known as Pilgrim America Prime Rate Trust), a Massachusetts business
trust, as successor administrative agent ("Pilgrim" or the "New Administrative
                                           -------          ------------------
Agent").
-----

          WHEREAS, the Company, the Lenders and the Existing Administrative
Agent, are parties to an Amended and Restated Credit Agreement dated as of
January 26, 1998, amending and restating in its entirety the Credit Agreement
dated as of July 24, 1996 to which the Company, the Lenders and the Existing
Administrative Agent were parties (as so amended and restated, and as
subsequently amended and in effect on the date hereof, the "Credit Agreement");
                                                            ----------------

          WHEREAS, each of the Lenders other than Pilgrim (such Lenders being
herein referred to as the "Settling Lenders") are willing to acknowledge payment
                           ----------------
in full of the principal of and interest on their respective Loans, and all
other amounts owing to the Settling Lenders under the Credit Agreement, in
exchange for payment in cash to them of certain amounts;

          WHEREAS, Pilgrim is willing to consent to the Settling Lenders
receiving payment of such amounts without any requirement to share the proceeds
thereof with Pilgrim;

          WHEREAS, the parties hereto wish to provide for the New Administrative
Agent to succeed to the rights and responsibilities of the Existing
Administrative Agent under the Credit Agreement;

          WHEREAS, concurrently with the satisfaction of the conditions
precedent to effectiveness of this Agreement, the Company and Pilgrim are
entering into a separate amendment (or amendment and restatement) of the Credit
Agreement, to which the Settling Lenders will not be party, and pursuant to
which the Settling Lenders will not be entitled to share in any payments
received by Pilgrim;

          NOW THEREFORE, in consideration of these premises and the mutual and
dependent promises hereinafter set forth, the parties hereto agree as follows:

<PAGE>

                                       -2-

                                    ARTICLE I

                                  DEFINED TERMS

          Terms used but not defined herein shall have the respective meanings
ascribed to such terms in the Credit Agreement.

                                   ARTICLE II

                           PAYMENT TO SETTLING LENDERS

          Each of the Settling Lenders hereby acknowledges and agrees that, upon
the satisfaction of the conditions specified in Article VII hereof, such
Settling Lender shall have been paid in full the principal of and interest on
the Loans, and all other amounts owing to such Settling Lender under the Credit
Agreement (other than amounts, if any, in respect of the obligations that under
Section 11.07 thereof are stated to survive the payment in full of the Loans),
and that such Settling Lender shall no longer be entitled to the benefits of any
of the collateral security provided in the Loan Documents. Each of the Borrower
and Pilgrim acknowledge that, upon such effectiveness, none of the Settling
Lenders shall have any further or continuing obligations under the Credit
Agreement and that the Settling Lenders shall no longer be a party thereto (and
no longer have the right, or be required, to consent to any future amendments
(or amendments and restatements) thereof), although each of the Settling Lenders
shall continue to be entitled to the benefits of said Section 11.07.

                                   ARTICLE III

                       REPLACEMENT OF ADMINISTRATIVE AGENT

          Upon the satisfaction of the conditions specified in Article VII
hereof, Chase hereby resigns as Administrative Agent under the Credit Agreement,
assigns to Pilgrim all of its rights as Administrative Agent under the Credit
Agreement and the other Loan Documents (other than those rights of Chase, as
retiring Administrative Agent, that are stated to survive pursuant to the last
sentence of Section 10.08 of the Credit Agreement) and Pilgrim hereby assumes
all of the responsibilities of the Administrative Agent under the Credit
Agreement arising on or after such resignation and assignment. In connection
with the foregoing, Chase hereby agrees with the New Administrative Agent to
execute and deliver, at the expense of the Company, all such instruments of
assignment and all other documents as shall be reasonably requested by the New
Administrative Agent in order to assign to the New Administrative Agent for the
benefit of Pilgrim the Liens granted to the Administrative Agent pursuant to the
Security Documents. Pilgrim, by its signature hereto, accepts, effective upon
the satisfaction of the conditions specified in Article VII hereof, its
appointment as Administrative Agent for purposes of the Credit Agreement and the
Security Documents.

<PAGE>

                                       -3-

                                   ARTICLE IV

                                 MUTUAL RELEASE

          Upon the satisfaction of the conditions specified in Article VII
hereof, each of the Company (and, by its signature below, the Guarantor), and
the Settling Lenders and the Existing Administrative Agent, for themselves,
their officers, directors, employees, legal representatives, successors and
assigns, does hereby forever and completely release, remise, acquit and
discharge each of the Settling Lenders, the Existing Administrative Agent, the
Company and the Guarantor, as well as any of their respective officers,
directors, shareholders, employees, and legal representatives in each case from
and against any and all claims, liabilities, demands, obligations, rights,
damages, costs, expenses, and causes of action of any nature, kind, or character
(whether tort, contract, or statutory, past or future, known or unknown,
suspected or unsuspected, fixed or contingent) which are based on, relate to,
arise from, or are in any way connected with any oral or written act, omission,
communication, representation, warranty, covenant, agreement, or course of
conduct occurring on or at any time prior to the date hereof and related to the
Credit Agreement, except for claims arising from obligations under those
provisions of the Credit Agreement that by their terms expressly state that they
survive termination of the Credit Agreement.

                                    ARTICLE V

                        CONSENT AMONG LENDERS TO PAYMENT

          Pilgrim hereby consents to each of the Settling Lenders being paid the
amounts indicated opposite their names on Schedule I hereto and agrees that none
of the Settling Lenders shall, notwithstanding the provisions of Section 4.07 of
the Credit Agreement or any other provision of any of the Loan Documents, be
required to share with Pilgrim any of such amounts (by purchase of
participations or assignments or otherwise). Each of the Settling Lenders hereby
consents to Pilgrim being paid the amount indicated opposite its name on
Schedule I hereto and agrees that Pilgrim shall not, notwithstanding the
provisions of Section 4.07 of the Credit Agreement or any other provision of any
of the Loan Documents, be required to share with the Settling Lenders any of
such amounts or any amounts received by Pilgrim pursuant to any amendment (or
amendment and restatement) of the Credit Agreement entered into by Pilgrim and
the Company contemporaneously with or subsequent to the date of this Agreement.

          Each of the Lenders hereby represents and warrants to each of the
other Lenders that such Lender is not receiving any payment or other
consideration from the Company or any of its Subsidiaries or Affiliates in
respect of the execution and delivery of this Agreement other than the cash
payments identified opposite its name in Schedule I hereto and, in the case of
Pilgrim, (i) the restructuring of the remaining balance of its outstanding Loans
under the Credit Agreement into a Loan having a remaining principal balance of
$2,750,000 and (ii) a fee in the amount of $50,000.

<PAGE>

                                       -4-

                                   ARTICLE VI

                          AMENDMENT TO CREDIT AGREEMENT

          The Company and Pilgrim hereby acknowledge that, concurrently with the
satisfaction of the conditions precedent set forth in Article VII hereof, they
are entering into a separate amendment (or amendment and restatement) of the
Credit Agreement reflecting, inter alia, that the Settling Lenders are no longer
                             ----- ----
party thereto and that Pilgrim is the Administrative Agent thereunder. Nothing
in any such amendment (or amendment and restatement), or any other document or
agreement that may at any time be executed by the Company and Pilgrim, shall
alter or impair the obligations of the Company that are stated to survive
pursuant to Article II and Article III hereof for the benefit of the Settling
Lenders and the Administrative Agent.

                                   ARTICLE VII

                                  EFFECTIVENESS

          The acknowledgment of the Settling Lenders set forth in Article II
hereof, the replacement of the Administrative Agent set forth in Article III
hereof and the mutual releases set forth in Article IV hereof shall become
effective upon (i) the execution of this Agreement by the Company, each Lender,
the Existing Administrative Agent and the New Administrative Agent; (ii) receipt
by the Administrative Agent, for account of each of the Settling Lenders, of the
amount indicated opposite the name of such Lender in Schedule I hereto, in
immediately available funds; (iii) receipt by Pilgrim, for its own account, of
the amount indicated opposite its name in Schedule I hereto, in immediately
available funds; (iv) the payment of the fees and expenses of Milbank, Tweed,
Hadley & McCloy LLP, special New York counsel to the Existing Administrative
Agent in the amount separately specified to the Company; and (v) receipt by the
Administrative Agent of written confirmation from Pilgrim that the conditions
precedent to the effectiveness of the separate amendment (or amendment and
restatement) of the Credit Agreement described in Article VI have either been
waived or fulfilled to Pilgrim's satisfaction.

                                  ARTICLE VIII

                                  MISCELLANEOUS

          Except as expressly herein provided, the Credit Agreement shall remain
unchanged and in full force and effect. This Agreement may be executed in any
number of counterparts, all of which taken together shall constitute one and the
same amendatory instrument and any of the parties hereto may execute this
Agreement by signing any such counterpart. This Agreement shall be governed by,
and construed in accordance with, the law of the State of New York.

<PAGE>

                                      -5-

          IN WITNESS WHEREOF, the parties hereto have caused this Agreement to
be duly executed as of the day and year first above written.

                               TELETOUCH COMMUNICATIONS, INC.



                               By /s/ J. Kernan Crotty
                                 -------------------------------------------
                                 Title: President

                               JPMORGAN CHASE BANK (successor to
                                The Chase Manhattan Bank), as Existing
                                Administrative Agent



                               By___________________________________________
                                 Title:

                               ING PRIME RATE TRUST,
                               as assignee and New Administrative Agent

                                By ING Pilgrim Investments,
                                 as its investment manager



                               By___________________________________________
                                 Title:

<PAGE>

                                      -5-

          IN WITNESS WHEREOF, the parties hereto have caused this Agreement to
be duly executed as of the day and year first above written.

                               TELETOUCH COMMUNICATIONS, INC.



                               By___________________________________________
                                 Title:

                               JPMORGAN CHASE BANK (successor to
                                The Chase Manhattan Bank), as Existing
                                Administrative Agent



                               By /s/ Patrick A. Daniello
                                 -------------------------------------------
                                 Title: PATRICK DANIELLO
                                        MANAGING DIRECTOR

                               ING PRIME RATE TRUST,
                               as assignee and New Administrative Agent

                                By ING Pilgrim Investments,
                                 as its investment manager



                               By___________________________________________
                                 Title:

<PAGE>

                                      -5-

          IN WITNESS WHEREOF, the parties hereto have caused this Agreement to
be duly executed as of the day and year first above written.

                                      TELETOUCH COMMUNICATIONS, INC.

                                      By /s/ J. Kernan Crotty
                                         ------------------------------------
                                         Title: President

                                      JPMORGAN CHASE BANK (successor to The
                                       Chase Manhattan Bank), as Existing
                                       Administrative Agent

                                      By ____________________________________
                                         Title:

                                      ING PRIME RATE TRUST, as assignee and New
                                      Administrative Agent

                                      By ING Pilgrim Investments, as its
                                       investment manager

                                      By /s/ Robert Wilson
                                         ------------------------------------
                                         Title:

<PAGE>

                                      -6-

                                        LENDERS
                                        -------

                                        JPMORGAN CHASE BANK (successor to The
                                         Chase Manhattan Bank)


                                        By /s/  Patrick A. Daniello
                                          ------------------------------------
                                          Title: Managing Director

                                        FINOVA CAPITAL


                                        By____________________________________
                                          Title:

                                        VAN KAMPEN PRIME RATE INCOME TRUST
                                         By Van Kampen Investment Advisory Corp.


                                        By____________________________________
                                          Title:

                                        ING PRIME RATE TRUST,
                                         By ING Pilgrim Investments, as its
                                          investment manager


                                        By____________________________________
                                          Title:

<PAGE>

                                      -6-

                                        LENDERS
                                        -------

                                        JPMORGAN CHASE BANK (successor to
                                         The Chase Manhattan Bank)



                                        BY______________________________________
                                          Title:

                                        FINOVA CAPITAL CORPORATION


                                        By /s/ [ILLEGIBLE]
                                          --------------------------------------
                                          Title: Vice President

                                        VAN KAMPEN PRIME RATE INCOME
                                        TRUST
                                         By Van Kampen Investment Advisory Corp.



                                        By______________________________________
                                          Title


                                        ING PRIME RATE TRUST,
                                         By ING Pilgrim Investments,
                                         as its investment manager



                                        By______________________________________
                                          Title:

<PAGE>

                                      -6-

                                        LENDERS
                                        -------

                                        JPMORGAN CHASE BANK (successor to
                                         The Chase Manhattan Bank)



                                        BY______________________________________
                                          Title:

                                        FINOVA CAPITAL


                                        By______________________________________
                                          Title:

                                        VAN KAMPEN PRIME RATE INCOME
                                        TRUST
                                         By Van Kampen Investment Advisory Corp.



                                        By /s/ Darvin D. Pierce
                                          --------------------------------------
                                          Title: Executive Director


                                        ING PRIME RATE TRUST,
                                         By ING Pilgrim Investments,
                                         as its investment manager



                                        By______________________________________
                                          Title:

<PAGE>

                                      -6-

                                        LENDERS
                                        -------

                                        JPMORGAN CHASE BANK (successor to
                                         The Chase Manhattan Bank)



                                        BY______________________________________
                                          Title:

                                        FINOVA CAPITAL


                                        By _____________________________________
                                          Title:

                                        VAN KAMPEN PRIME RATE INCOME
                                        TRUST
                                         By Van Kampen Investment Advisory Corp.



                                        By______________________________________
                                          Title:


                                        ING PRIME RATE TRUST,
                                         By ING Pilgrim Investments,
                                         as its investment manager



                                        By /s/ Robert Wilson
                                          --------------------------------------
                                          Title:

<PAGE>

                                      -7-

                                   GUARANTOR
                                   ---------

          The undersigned, the sole Guarantor under the Guarantee and Security
Agreement, hereby consents to the execution, delivery and performance of this
Agreement for all purposes of the Guarantee and Security Agreement and agrees to
the release set forth in Article IV above.

TELETOUCH LICENSES, INC.

By /s/ J. Kernan Crotty
  -----------------------
Title: President

<PAGE>

                                  Schedule I to
                 Partial Payoff and Agent Replacement Agreement


   Name of Settling Lender                                       Amount
   -----------------------                                       ------

   JPMorgan Chase Bank                                       $4,202,316.98

   Finova Capital                                            $3,179,425.31

   Van Kampen Prime Rate Income Trust                        $1,825,095.22


   Pilgrim
   -------

   Pilgrim America Prime Rate Trust                           $ 543,160.00