<PAGE>   1
                                                                  EXHIBIT 10.4.4


                       AMENDMENT NO. 4 TO CREDIT AGREEMENT

                  AMENDMENT NO. 4, dated as of June 30, 2001 (this "Amendment")
to Credit Agreement, by and among Private Business, Inc., a Tennessee
corporation (the "Borrower"), the Lenders and Fleet National Bank, as the
Initial Issuing Bank, the Swing Line Bank and the Administrative Agent.

                             PRELIMINARY STATEMENTS

                  (A) The Borrower, the Lenders and Fleet National Bank, as the
Initial Issuing Bank, the Swing Line Bank and the Administrative Agent, are
parties to the Credit Agreement, dated as of August 7, 1998 (as amended to the
date hereof, the "Credit Agreement").

                  (B) The Borrower has requested that the Lenders consent to (i)
the merger (the "Towne Merger") of Towne Acquisition Corporation, a Tennessee
corporation and a wholly-owned subsidiary of the Borrower ("Merger Sub"), with
and into Towne Services, Inc., a Georgia corporation ("Towne"), and (ii) the
issuance of up to 40,032 shares of Series B Preferred Stock of the Borrower (the
"Series B Preferred Stock") in connection with the Towne Merger.

                  (C) The Borrower has requested that the Lenders amend the
Credit Agreement as more fully set forth below.

                  (D) The Administrative Agent and the Lenders are willing to
consent to the Towne Merger and the issuance of up to 40,032 shares of Series B
Preferred Stock and to amend the Credit Agreement on the terms and conditions
set forth herein.

                  (E) The terms defined in the Credit Agreement and not
otherwise defined herein shall have the meanings ascribed to them in the Credit
Agreement.

                  NOW, THEREFORE, in consideration of the premises and of the
mutual covenants and agreements contained herein, the parties hereto hereby
agree as follows:

ARTICLE 1. Consents to Towne Merger.

         Section 1.1. Notwithstanding the provisions of Section 6.4 of the
Credit Agreement, the Lenders hereby consent to the Towne Merger.

         Section 1.2. Notwithstanding the provisions of Section 6.19 of the
Credit Agreement, the Lenders hereby consent to the issuance by the Borrower of
up to 40,032 shares of Series B Preferred Stock on the terms and conditions set
forth in the Designations of Preferences, Limitations and Relative Rights
attached as Exhibit A hereto.

         Section 1.3. The consents set forth in this Section 1 are subject to
the satisfaction of each of the conditions precedent set forth in Section 5 of
this Amendment and shall be limited precisely as written and shall not be deemed
to (a) be a waiver, amendment or modification of any term or condition of the
Credit Agreement or any other Loan Document, except as set forth in this Section
1 (subject to the terms and conditions set forth herein), or (b) prejudice any
right, power or remedy which the Administrative Agent or any Lender Party now
has or may have in the future under or in connection with the Credit Agreement
or any other Loan Document.


<PAGE>   2

ARTICLE 2. Amendments to Credit Agreement.

         Section 2.1. Amendment. This Amendment shall be deemed to be an
amendment to the Credit Agreement and shall not be construed in any way as a
replacement or substitution therefor. All of the terms and conditions of, and
terms defined in, this Amendment are hereby incorporated by reference into the
Credit Agreement as if such terms and provisions were set forth in full therein.

         Section 2.2. Amendments to Credit Agreement. Effective upon the
satisfaction of the conditions precedent set forth in Article 4 hereof, the
Credit Agreement is hereby amended as follows:

                  (a) The aggregate amount of the Revolving Credit Commitments
         are reduced from $10,000,000 to $7,500,000. The Revolving Credit
         Commitment of each Revolving Credit Lender shall be reduced pro rata
         such that the new Revolving Credit Commitment of each Revolving Credit
         Lender shall be as set forth below:

<TABLE>
<CAPTION>
                                                                   Revolving
                  Revolving Credit Lender                      Credit Commitment
                  -----------------------                      -----------------
<S>                                                            <C>
                  Fleet National Bank                            $3,409,090.91

                  Citizens Bank of Massachusetts                 $2,045,454.55

                  LaSalle Bank National Association              $1,363,636.37

                  Credit Lyonnais New York Branch                  $681,818.18
</TABLE>

                  (b) Section 1.1 is amended by inserting the following as a new
         clause (vii) in the definition of EBITDA after existing clause (vi):
         "and (vii) costs and expenses related to the merger of Towne
         Acquisition Corporation, a wholly-owned Subsidiary of Borrower, with
         Towne Services, Inc. up to an amount not to exceed $5,500,000".

                  (c) Section 6.17 is amended by deleting the maximum amount of
         Capital Expenditures for Fiscal Year 2001 and replacing it with the
         amount "$3,500,000".

                  (d) Section 8.1 is amended by deleting the minimum EBITDA
         covenant levels for the period ending on or about June 30, 2001 and all
         periods thereafter and replacing them with the following:

<TABLE>
<CAPTION>
         Four Fiscal Quarters ending on or about:      Minimum EBITDA
         ----------------------------------------      --------------
<S>                                                    <C>
         June 30, 2001                                   $13,750,000
         September 30, 2001                              $11,500,000
         December 31, 2001                               $12,500,000

         March 31, 2002                                  $15,000,000
         June 30, 2002                                   $16,000,000
         September 30, 2002                              $16,500,000
         December 31, 2002                               $16,500,000
</TABLE>



                                      -2-
<PAGE>   3

<TABLE>
<S>                                                    <C>
         March 31, 2003                                  $16,500,000
         June 30, 2003                                   $16,500,000
         September 30, 2003                              $17,000,000
         December 31, 2003                               $17,000,000

         March 31, 2004                                  $17,000,000
         June 30, 2004                                   $17,500,000
         September 30, 2004                              $17,500,000
         December 31, 2004                               $18,000,000

         March 31, 2005                                  $18,000,000
         June 30, 2005                                   $18,000,000
         September 30, 2005                              $18,500,000
         December 31, 2005                               $18,500,000

         March 31, 2006                                  $18,500,000
         June 30, 2006 and thereafter                    $20,000,000
</TABLE>

                  (e) Section 8.2 is amended by deleting the maximum Ratio of
         Consolidated Debt to EBITDA covenant levels for the periods ending on
         or about June 30, 2001, September 30, 2001 and December 31, 2001 and
         replacing them with the following:

<TABLE>
<CAPTION>
         Four Fiscal Quarters ending on or about:             Ratio
         ----------------------------------------             -----
<S>                                                           <C>
         June 30, 2001                                        3.25:1
         September 30, 2001                                   3.00:1
         December 31, 2001                                    2.75:1
</TABLE>

                  (f) Section 8.3 is amended by deleting the minimum Interest
         Coverage Ratio covenant levels for the periods ending on or about June
         30, 2001 and September 30, 2001 and replacing them with the following:

<TABLE>
<CAPTION>
         Four Fiscal Quarters ending on or about:             Ratio
         ----------------------------------------             -----
<S>                                                           <C>
         June 30, 2001                                        3.00:1
         September 30, 2001                                   3.00:1
</TABLE>

ARTICLE 3. Confirmations and References.

         Section 3.1. Continuing Effect. The Credit Agreement and the other Loan
Documents delivered in connection therewith are, and shall continue to be, in
full force and effect, and are hereby ratified and confirmed in all respects,
except that, on and after the date hereof, (a) all references in the Loan
Documents (i) to the "Credit Agreement," "thereto," "thereof," "thereunder" or
words of like import referring to the Credit Agreement shall mean the Credit
Agreement as amended hereby and (ii) to the "Loan Documents" shall be deemed to
include this Amendment; and (b) all references in the Credit Agreement to "this
Agreement," "hereto," "hereof," "hereunder" or words of like import referring to
the Credit Agreement shall mean the Credit Agreement as amended hereby.



                                      -3-
<PAGE>   4

         Section 3.2. Confirmation of Liens. The Liens granted pursuant to the
Collateral Documents secure, without limitation, the Obligations of the Borrower
and its Subsidiaries to the Lenders and the Administrative Agent under the
Credit Agreement as amended by this Amendment. The term "Obligations" as used in
the Collateral Documents (or any other term used therein to refer to the
liabilities and obligations of the Borrower and its Subsidiaries to the Lenders
and the Administrative Agent), include, without limitation, Obligations to the
Lenders and the Administrative Agent under the Credit Agreement as amended by
this Amendment.

ARTICLE 4. Representations and Warranties.

         The Borrower and each Loan Party hereby represent and warrant to the
Lenders and the Administrative Agent that:

         Section 4.1. Existing Representations. Each of the representations and
warranties contained in Article 4 of the Credit Agreement is true in all
material respects on, and as though made as of, the date hereof, other than any
such representation or warranty that, by its terms, refers to a specific date,
in which case, as of such specific date.

         Section 4.2. No Default. As of the date hereof, there exists no Default
or Event of Default under the Credit Agreement, as amended hereby, and no event
which, with the giving of notice or lapse of time, or both, would constitute a
Default or Event of Default.

         Section 4.3. Power, Authority, Consents. The Borrower and each of its
Subsidiaries has the power to execute, deliver and perform the Credit Agreement,
as amended by this Amendment and to consummate each of the transactions
consented to in Section 1 hereof. The Borrower and each of its Subsidiaries has
taken all necessary action to authorize the execution, delivery and performance
of this Amendment and the consummation of each of the transactions consented to
in Section 1 hereof. No consent or approval of any Person is required in
connection with the execution, delivery or performance by the Borrower or any of
its Subsidiaries of this Amendment or the consummation of any of the
transactions consented to in Section 1 hereof.

         Section 4.4. No Violation of Law or Agreements. The execution, delivery
and performance by the Borrower and each of its Subsidiaries of this Amendment
and the consummation of each of the transactions consented to in Section 1
hereof, will not violate any provision of law presently in effect and will not
conflict with or result in a breach of any order, writ, injunction, ordinance,
resolution, decree, or other similar document or instrument presently in effect
of any court or governmental authority, bureau or agency, domestic or foreign,
or the certificate of incorporation or by-laws of the Borrower or such
Subsidiary, or create (with or without the giving of notice or lapse of time, or
both) a default under or breach of any agreement, bond, note or indenture
presently in effect to which the Borrower or any Subsidiary is a party, or by
which any of them is bound or any of their properties or assets is affected, or
result in the imposition of any Lien of any nature whatsoever upon any of the
properties or assets owned by or used in connection with the business of the
Borrower or any of its Subsidiaries, except for the Liens created and granted
pursuant to the Collateral Documents as acknowledged and confirmed herein.

         Section 4.5. Binding Effect. This Amendment has been duly executed and
delivered by the Borrower and each of its Subsidiaries and constitutes the valid
and legally binding



                                      -4-
<PAGE>   5

obligation of the Borrower and each of its Subsidiaries, except as such
enforceability may be limited by applicable bankruptcy, insolvency,
reorganization, moratorium or other similar laws, now or hereafter in effect,
relating to or affecting the enforcement of creditors' rights generally, and
except that the remedy of specific performance and other equitable remedies are
subject to judicial discretion.

ARTICLE 5. Conditions to Amendment.

         The effectiveness of the amendments contained in Article 1 shall be
subject to the fulfillment of the following conditions precedent:

         Section 5.1. Amendment. The Borrower, the Required Lenders and
Revolving Credit Lenders holding greater than 50% of the aggregate Revolving
Credit Commitments shall have executed and delivered to the Administrative Agent
this Amendment.

         Section 5.2. No Default. There shall exist no Event of Default or
Default under the Credit Agreement.

         Section 5.3. Representations and Warranties. The representations and
warranties contained in Article 4 hereof shall be true and correct in all
material respects on the date hereof.

         Section 5.4. Amendment Fee. The Borrower shall have paid an amendment
fee to the Administrative Agent, for the account of each Lender which has
approved this Amendment, as evidenced by such Lender's timely execution and
delivery of a counterpart signature page to this Amendment, in an amount equal
to 0.50% (i.e. 50 basis points) of the aggregate of such approving Lenders'
Commitments immediately after the effectiveness of this Amendment.

         Section 5.5. Towne Merger Conditions and Deliveries.

                  (a) Merger Sub shall have executed and delivered a Subsidiary
         Guaranty Supplement, a Security Agreement Supplement, an Intellectual
         Property Security Agreement Supplement, all such documents to be in
         form and substance satisfactory to the Administrative Agent.

                  (b) The Borrower shall have delivered to the Administrative
         Agent copies of all legal opinions delivered by counsel to the Borrower
         or to Towne in connection with the Towne Merger, in form and substance
         satisfactory to the Administrative Agent, and the Administrative Agent
         and the Lenders shall be permitted to rely on such opinions.

                  (c) The Borrower shall have delivered to the Administrative
         Agent executed copies of the Agreement and Plan of Merger among the
         Borrower, Merger Sub and Towne, dated April 12, 2001, along with all
         exhibits and disclosure schedules thereto, and all amendments thereto,
         all of which shall be in form and substance satisfactory to the
         Administrative Agent.

                  (d) The Borrower shall have delivered to the Administrative
         Agent a copy of the filed Designations of Preferences, Limitations and
         Relative Rights of the Series B Preferred Stock, certified by the
         Secretary of State of the State of Tennessee.



                                      -5-
<PAGE>   6

ARTICLE 6. Post-Closing Covenants.

         Section 6.1. Covenants. The Borrower hereby agrees to deliver to the
Administrative Agent the following documents, in form and substance satisfactory
to the Administrative Agent, and/or to take the following actions in a manner
acceptable to the Administrative Agent:

                  (a) On the second Business Day after the effective date of the
         Towne Merger, the Borrower shall prepay the Advances in an amount of at
         least $6,000,000 and, notwithstanding the provisions of Section 2.6(a)
         of the Credit Agreement, such prepayment shall be applied solely to
         scheduled principal installments under the Term B Facility in inverse
         order of maturity.

                  (b) Immediately after the effectiveness of the Towne Merger,
         Towne and each of Towne's Subsidiaries shall execute and deliver to the
         Administrative Agent a Subsidiary Guaranty Supplement, a Security
         Agreement Supplement, an Intellectual Property Security Agreement
         Supplement and such other Collateral Documents as the Administrative
         Agent shall request in order to grant to the Administrative Agent, for
         its benefit and the ratable benefit of the Secured Parties, a perfected
         first priority security interest in all personal property of each such
         company (including, without limitation, all stock or other equity
         interests of each Subsidiary of each such company).

                  (c) Within 5 Business Days after the effectiveness of the
         Towne Merger, the Borrower shall deliver to the Administrative Agent
         stock certificates representing all of the issued and outstanding
         shares of Towne and each of Towne's Subsidiaries, along with an undated
         stock power duly executed in blank for each such stock certificate.

                  (d) Within 60 days after the effectiveness of the Towne
         Merger, the Borrower, Towne and each of their respective Subsidiaries
         shall execute and deliver a Mortgage with respect to each parcel of
         real property owned by any of them creating a valid and enforceable
         first priority lien on each such parcel in favor of the Administrative
         Agent for its benefit and the ratable benefit of the Secured Parties,
         together with such mortgagee title insurance policies, surveys, local
         counsel opinions and other documents, certificates or instruments which
         shall be reasonably requested by the Administrative Agent, all in form
         and substance satisfactory to the Administrative Agent.

         Section 6.2. Event of Default. The failure of the Borrower or any of
its Subsidiaries (including, without limitation, Towne and each of its
Subsidiaries) to perform or comply with any of the covenants set forth in this
Article 6 within the time periods set forth herein shall automatically be an
Event of Default under the Credit Agreement.

ARTICLE 7. Miscellaneous.

         Section 7.1. Continued Effectiveness. Except as specifically amended
herein, the Credit Agreement and each of the other Loan Documents shall remain
in full force and effect in accordance with their respective terms.

         Section 7.2. Governing Law. This Amendment shall be governed and
construed in accordance with the laws of the State of New York.



                                      -6-
<PAGE>   7

         Section 7.3. Severability. The provisions of this Amendment are
severable, and if any clause or provision shall be held invalid or unenforceable
in whole or in part in any jurisdiction, then such invalidity or
unenforceability shall affect only such clause or provision, or part thereof, in
such jurisdiction and shall not in any manner affect such clause or provision in
any other jurisdiction, or any other clause or provision in this Amendment in
any jurisdiction.

         Section 7.4. Counterparts. This Amendment may be signed in any number
of counterparts with the same effect as if the signatures thereto and hereto
were upon the same instrument. Delivery of an executed counterpart of this
Amendment by facsimile shall be as effective as delivery of an originally
executed counterpart.

         Section 7.5. Binding Effect; Assignment. This Amendment shall be
binding upon and inure to the benefit of the Borrower and its respective
successors and to the benefit of the Administrative Agent and the Lenders and
their respective successors and assigns. The rights and obligations of the
Borrower under this Amendment shall not be assigned or delegated without the
prior written consent of the Lenders, and any purported assignment or delegation
without such consent shall be void.

         Section 7.6. Expenses. The Borrower shall pay the Administrative Agent
upon demand for all reasonable expenses, including reasonable fees of counsel
for the Administrative Agent, incurred by the Administrative Agent in connection
with the preparation, negotiation and execution of this Amendment and any
documents required to be furnished herewith.



                            [Signature Pages Follow]


<PAGE>   8

                  IN WITNESS WHEREOF, the parties hereto have caused this
Amendment to be executed by their respective officers thereunto duly authorized
on the date first above written.


                                    PRIVATE BUSINESS, INC.,
                                    AS BORROWER



                                    By: /s/ Fred P. Read
                                        ----------------------------------------
                                    Title: Chief Financial Officer
                                           -------------------------------------


                                    ACKNOWLEDGED AND CONSENTED TO:

                                    PRIVATE BUSINESS INSURANCE, INC.



                                    By: /s/ Fred P. Read
                                        ----------------------------------------
                                    Title: Chief Financial Officer
                                           -------------------------------------



                                    PRIVATE BUSINESS CAPITAL, INC.



                                    By: /s/ Fred P. Read
                                        ----------------------------------------
                                    Title: Chief Financial Officer
                                           -------------------------------------



                                    PRIVATE BUSINESS PROCESSING, INC.



                                    By: /s/ Fred P. Read
                                        ----------------------------------------
                                    Title: Chief Financial Officer
                                           -------------------------------------



                                    TOWNE ACQUISITION CORPORATION



                                    By: /s/ Fred P. Read
                                        ----------------------------------------
                                    Title: Chief Financial Officer
                                           -------------------------------------


<PAGE>   9

                                    FLEET NATIONAL BANK,
                                    AS INITIAL ISSUING BANK,
                                    AS SWING LINE BANK,
                                    AS ADMINISTRATIVE AGENT AND
                                    AS LENDER



                                    By: /s/ Stephen M. Curran
                                        ----------------------------------------
                                    Title: Director
                                           -------------------------------------


<PAGE>   10

                                    CITIZENS BANK OF MASSACHUSETTS,
                                    AS LENDER



                                    By: /s/ Lawrence E. Jacobs
                                        ----------------------------------------
                                    Title: Vice President
                                           -------------------------------------


<PAGE>   11

                                    PILGRIM PRIME RATE TRUST,
                                    AS LENDER

                                    By: Pilgrim Investments, Inc.,
                                    as its Investment Manager




                                    By: /s/ Michael Prince
                                        ----------------------------------------
                                    Title: Vice President
                                           -------------------------------------

<PAGE>   12

                                    LASALLE BANK NATIONAL ASSOCIATION,
                                    AS LENDER



                                    By: /s/ Todd M. Kostelnik
                                        ----------------------------------------
                                    Title: Corporate Banking Officer
                                           -------------------------------------


<PAGE>   13

                                    CREDIT LYONNAIS NEW YORK BRANCH,
                                    AS LENDER



                                    By:
                                        ----------------------------------------
                                    Title:
                                           -------------------------------------