================================================================================

                               GUARANTEE AGREEMENT

                                     Between

                            U.S.B. HOLDING CO., INC.
                                 (as Guarantor)

                                       and

                            THE CHASE MANHATTAN BANK
                                  (as Trustee)

                                   Dated as of

                                February 5, 1997

================================================================================
<PAGE>

                                TABLE OF CONTENTS

                                                                           Page
                                                                           ----

                                    ARTICLE I


                                   Definitions

   SECTION 1.01. Definitions.................................................2

                                   ARTICLE II

                               Trust Indenture Act

   SECTION 2.01. Trust Indenture Act; Application............................4
   SECTION 2.02. List of Holders.............................................5
   SECTION 2.03. Reports by the Guarantee Trustee............................5
   SECTION 2.04. Periodic Reports to the Guarantee Trustee...................5
   SECTION 2.05. Evidence of Compliance with Conditions Precedent............5
   SECTION 2.06. Events of Default, Waiver...................................5
   SECTION 2.07. Event of Default; Notice....................................6
   SECTION 2.08. Conflicting Interests.......................................6

                                   ARTICLE III

              Powers, Duties and Rights of the Guarantee Trustee

   SECTION 3.01. Powers and Duties of the Guarantee Trustee..................6
   SECTION 3.02. Certain Rights of Guarantee Trustee.........................8
   SECTION 3.03. Indemnity...................................................9
   SECTION 3.04. Expenses....................................................9

                                   ARTICLE IV

                                Guarantee Trustee

   SECTION 4.01. Guarantee Trustee; Eligibility.............................10
   SECTION 4.02. Appointment, Removal and Resignation of the Guarantee
                 Trustee....................................................10

                                    ARTICLE V

                                    Guarantee

   SECTION 5.01. Guarantee..................................................11
<PAGE>

                                TABLE OF CONTENTS

                                                                           Page
                                                                           ----

   SECTION 5.02. Waiver of Notice and Demand................................11
   SECTION 5.03. Obligations Not Affected...................................11
   SECTION 5.04. Rights of Holders..........................................12
   SECTION 5.05. Guarantee of Payment.......................................12
   SECTION 5.06. Subrogation................................................12
   SECTION 5.07. Independent Obligations....................................13

                                   ARTICLE VI

                           Covenants and Subordination

   SECTION 6.01. Subordination..............................................13
   SECTION 6.02. Pari Passu Guarantees......................................13

                                   ARTICLE VII

                                   Termination

   SECTION 7.01. Termination................................................15

                                  ARTICLE VIII

                                  Miscellaneous

   SECTION 8.01. Successors and Assigns.....................................15
   SECTION 8.02. Amendments.................................................15
   SECTION 8.03. Notices....................................................15
   SECTION 8.04. Benefit....................................................16
   SECTION 8.05. Interpretation.............................................17
   SECTION 8.06. Governing Law..............................................17
<PAGE>

                             CROSS-REFERENCE TABLE*

Section of
Trust Indenture Act                                             Section of
of 1939, as amended                                         Guarantee Agreement
-------------------                                         -------------------

310 (a)   ..................................................      4.01 (a)
310 (b)   ..................................................      4.01 (c), 2.08
310 (c)   ..................................................  Inapplicable
311 (a)   ..................................................      2.02 (b)
311 (b)   ..................................................      2.02 (b)
311 (c)   ..................................................  Inapplicable
312 (a)   ..................................................      2.02 (a)
312 (b)   ..................................................      2.02 (b)
313.      ..................................................      2.03
314 (a)   ..................................................      2.04
314 (b)   ..................................................  Inapplicable
314 (c)   ..................................................      2.05
314 (d)   ..................................................  Inapplicable
314 (e)   ..................................................      1.01, 2.05,
                                                                  3.02
314 (f)   ..................................................      2.01, 3.02
315 (a)   ..................................................      3.01 (d)
315 (b)   ..................................................      2.07
315 (c)   ..................................................      3.01
315 (d)   ..................................................      3.01 (d)
316 (a)   ..................................................      1.01, 2.06,
                                                                  5.04
316 (b)   ..................................................      5.03
316 (c)   ..................................................  Inapplicable
317 (a)   ..................................................  Inapplicable
317 (b)   ..................................................  Inapplicable
318 (a)   ..................................................      2.01 (b)


----------
* This Cross-Reference Table does not constitute part of the Guarantee Agreement
and shall not affect the interpretation of any of its terms or provisions.
<PAGE>

            GUARANTEE AGREEMENT, dated as of February 5, 1997, executed and
delivered by U.S.B. HOLDING CO., INC., a Delaware corporation (the "Guarantor")
having its principal office at 100 Dutch Hill Road, Orangeburg, New York 10962,
and THE CHASE MANHATTAN BANK, a New York banking corporation, as trustee (the
"Guarantee Trustee"), for the benefit of the Holders (as defined herein) from
time to time of the Trust Securities (as defined herein) of UNION STATE CAPITAL
TRUST I, a Delaware statutory business trust (the "Issuer").

            WHEREAS pursuant to an Amended and Restated Declaration of Trust
(the "Declaration of Trust"), dated as of February 5, 1997, among the Trustees
named therein, the Guarantor, as Depositor, and the Holders from time to time of
undivided beneficial ownership interests in the assets of the Issuer, the Issuer
is issuing $20,000,000 aggregate liquidation amount of its 9.58% Capital
Securities, liquidation amount $1,000 per Security (the "Capital Securities")
and $619,000 of aggregate liquidation amount of 9.58% Common Securities,
liquidation amount $1,000 per security (the "Common Securities" and collectively
with the Capital Securities, the "Trust Securities") representing undivided
beneficial ownership interests in the assets of the Issuer and having the terms
set forth in the Declaration of Trust;

            WHEREAS the Trust Securities will be issued by the Issuer and the
proceeds thereof will be used by the Issuer to purchase $20,619,000 aggregate
principal amount of the 9.58% Junior Subordinated Debt Securities due February
1, 2027 (the "Junior Subordinated Debt Securities") of the Guarantor, which will
be held by The Chase Manhattan Bank, as Property Trustee under the Declaration
of Trust, as trust assets; and

            WHEREAS as incentive for the Holders to purchase Trust Securities
the Guarantor desires irrevocably and unconditionally to agree, to the extent
set forth herein, to pay to the Holders the Guarantee Payments (as defined
herein) and to make certain other payments on the terms and conditions set forth
herein.

            NOW, THEREFORE, in consideration of the purchase by each Holder,
which purchase the Guarantor hereby agrees shall benefit the Guarantor, the
Guarantor executes and delivers this Guarantee Agreement for the benefit of the
Holders from time to time of the Trust Securities.
<PAGE>

                                    ARTICLE I

                                   Definitions

            SECTION 1.01. Definitions. As used in this Guarantee Agreement, the
terms set forth below shall, unless the context otherwise requires, have the
following meanings. Capitalized or otherwise defined terms used but not defined
herein shall have the meanings assigned to such terms in the Declaration of
Trust as in effect on the date hereof.

            "Affiliate" of any specified Person means any other Person directly
or indirectly controlling or controlled by or under direct or indirect common
control with such specified Person; provided, however, that an Affiliate of the
Guarantor shall not be deemed to include the Issuer. For the purposes of this
definition, "control" when used with respect to any specified Person means the
power to direct the management and policies of such Person, directly or
indirectly, whether through the ownership of voting securities, by contract or
otherwise; and the terms "controlling" and "controlled" have meanings
correlative to the foregoing.

            "Capital Securities" shall have the meaning specified in the first
recital of this Guarantee Agreement.

            "Common Securities" shall have the meaning specified in the first
recital of this Guarantee Agreement.

            "Debt" shall have the meaning specified in the Indenture.

            "Declaration of Trust" shall have the meaning specified in the first
recital of this Guarantee Agreement.

            "Event of Default" means a default by the Guarantor on any of its
payment or other obligations under this Guarantee Agreement; provided, however,
that, except with respect to a default in payment of any Guarantee Payments, the
Guarantor shall have received notice of default and shall not have cured such
default within 60 days after receipt of such notice; proved, further, that no
Event of Default shall be deemed to have occured unless an Event of Default (as
defined in the Indenture or the Declaration) shall have occurred and be
continuing.

            "Guarantee Payments" means the following payments or distributions,
without duplication, with respect to the Trust Securities, to the extent not
paid or made by or on behalf of the Issuer: (i) any accrued and unpaid
Distributions required to be paid on the Trust Securities, to the extent the
Issuer shall have funds on hand available therefor at such time, (ii) the
redemption price, including all accrued and unpaid Distributions to the date of
redemption (the "Redemption Price"), with respect to the Trust Securities called
for redemption by the Issuer to the extent the Issuer shall have funds on hand
available therefor at such time, and (iii) upon a voluntary or involuntary
termination, winding-up or liquidation of the Issuer, unless Junior Subordinated
Debt Securities are distributed to the Holders, the lesser


                                      -2-
<PAGE>

of (a) the aggregate of the liquidation amount of $1,000 per Trust Security plus
accrued and unpaid Distributions to the date of payment to the extent the Issuer
shall have funds on hand available to make such payment at such time and (b) the
amount of assets of the Issuer remaining available for distribution to Holders
in liquidation of the Issuer (in either case, the "Liquidation Distribution").
If an Event of Default under the Declaration has occurred and is continuing, no
Guarantee Payments with respect to the Common Securities or any guarantee
payment under any Other Guarantees (as defined in the Indenture) with respect to
Common Securities of any other U.S.B.H. Capital Trust (as defined in the
Indenture), if any, shall be made until the Holders of Capital Securities shall
be paid in full the Guarantee Payments to which they are entitled under this
Guarantee. Subordination of Guarantee Payments on the Common Securities
following such an Event of Default under the Declaration shall be analogous to
the subordination of the Common Securities provided for in Section 4.03 of the
Declaration.

            "Guarantee Trustee" means The Chase Manhattan Bank, until a
Successor Guarantee Trustee has been appointed and has accepted such appointment
pursuant to the terms of this Guarantee Agreement and thereafter means each such
Successor Guarantee Trustee.

            "Guarantor" shall have the meaning specified in the first recital of
this Guarantee Agreement.

            "Holder" means any holder, as registered on the books and records of
the Issuer, of any Trust Securities; provided, however, that in determining
whether the holders of the requisite percentage of Trust Securities have given
any request, notice, consent or waiver hereunder, "Holder" shall not include the
Issuer, the Guarantor, the Guarantee Trustee, or any Affiliate of the Guarantor
or the Guarantee Trustee.

            "Indenture" means the Junior Subordinated Indenture dated as of
February 5, 1997, as supplemented and amended between the Guarantor and The
Chase Manhattan Bank, as trustee, relating to the issuance of the Junior
Subordinated Debt Securities.

            "Issuer" shall have the meaning specified in the first recital of
this Guarantee Agreement.

            "List of Holders" has the meaning specified in Section 2.02(a).

            "Majority in Liquidation Amount of the Securities" means a vote by
the Holder(s), voting separately as a class, of more than 50% of the aggregate
Liquidation Amount of all then Outstanding Trust Securities.

            "Officers' Certificate" means, with respect to any Person, a
certificate signed by the Chairman, the Chief Executive Officer, the President
or a Vice President, and by the Chief Financial Officer, the Treasurer, an
Assistant Treasurer, the Controller, the Secretary or an Assistant Secretary of
such Person, and delivered to the Guarantee Trustee. Any Officers'


                                      -3-
<PAGE>

Certificate delivered with respect to compliance with a condition or covenant
provided for in this Guarantee Agreement shall include:

            (a) a statement that each officer signing the Officers' Certificate
has read the covenant or condition and the definitions relating thereto;

            (b) a brief statement of the nature and scope of the examination or
investigation undertaken by each officer in rendering the Officers' Certificate;

            (c) statement that each officer has made such examination or
investigation as, in such officer's opinion, is necessary to enable such officer
to express an informed opinion as to whether or not such covenant or condition
has been complied with; and

            (d) a statement as to whether, in the opinion of each officer, such
condition or covenant has been complied with.

            "Other Debentures" shall have the meaning specified in the
Indenture.

            "Other Guarantees" shall have the meaning specified in the
Indenture.

            "Registration Rights Agreement" shall have the meaning specified in
Section 12.01 of the Indenture.

            "Responsible Officer" when used with respect to the Guarantee
Trustee means any officer of the Trustee with direct responsibility for the
administration of this Guarantee Agreement, and also means, with respect to a
particular matter, any other officer of the Trustee to whom such matter is
referred because of such officer's knowledge of and familiarity with the
particular subject.

            "Senior Debt" shall have the meaning specified in the Indenture.

            "Successor Guarantee Trustee" means a successor Guarantee Trustee
possessing the qualifications to act as Guarantee Trustee under Section 4.01.

            "Trust Securities" shall have the meaning specified in the first
recital of this Guarantee Agreement.

                                   ARTICLE II

                               Trust Indenture Act

            SECTION 2.01. Trust Indenture Act; Application. (a) This Guarantee
Agreement will not be qualified under the Trust Indenture Act except upon the
effectiveness of a registration statement with respect to this Guarantee
Agreement pursuant to a registration rights agreement as contemplated in Article
XII of the Indenture.


                                      -4-
<PAGE>

            (b) Upon qualification under the Trust Indenture Act as contemplated
in clause (a) above, if and to the extent that any provision of this Guarantee
Agreement limits, qualifies or conflicts with the duties imposed by Sections 310
to 317, inclusive, of the Trust Indenture Act, such imposed duties shall
control.

            SECTION 2.02. List of Holders. (a) The Guarantor shall furnish or
cause to be furnished to the Guarantee Trustee (i) semiannually, not more than
15 days after May 15 and November 15 or each year, a list, in such form as the
Guarantee Trustee may reasonably require, of the names and addresses of the
Holders ("List of Holder") as of a date not more than 15 days prior to the
delivery thereof, and (ii) at such other times as the Guarantee Trustee may
request in writing, within 30 days after the receipt by the Guarantor of any
such request, a List of Holders as of a date not more than 15 days prior to the
time such list is furnished, in each case to the extent such information is in
the possession or control of the Guarantor and is not identical to a previously
supplied list of Holders or has not otherwise been received by the Guarantee
Trustee in its capacity as such. The Guarantee Trustee may destroy any List of
Holders previously given to it on receipt of a new List of Holders.

            (b) The Guarantee Trustee shall comply with its obligations under
Section 311 (a), Section 311(b) and Section 312(b) of the Trust Indenture Act.

            SECTION 2.03. Reports by the Guarantee Trustee. Within 60 days of
May 15 of each calendar year, commencing with May 15, 1997, the Guarantee
Trustee shall provide to the Holders such reports, if any, as are required by
Section 313 of the Trust Indenture Act in the form and in the manner provided by
Section 313 of the Trust Indenture Act. The Guarantee Trustee shall also comply
with the requirements of Section 313(d) of the Trust Indenture Act.

            SECTION 2.04. Periodic Reports to the Guarantee Trustee. The
Guarantor shall provide to the Guarantee Trustee, the Securities and Exchange
Commission and the Holders such documents, reports and information, if any, as
required by Section 314 of the Trust Indenture Act and the compliance
certificate required by Section 314 of the Trust Indenture Act, in the form, in
the manner and at the times required by Section 314 of the Trust Indenture Act.

            SECTION 2.05. Evidence of Compliance with Conditions Precedent. The
Guarantor shall provide to the Guarantee Trustee such evidence of compliance
with such conditions precedent, if any, provided for in this Guarantee Agreement
that relate to any of the matters set forth in Section 314(c) of the Trust
Indenture Act. Any certificate or opinion required to be given by any officer
pursuant to Section 314 (c)(1) may be given in the form of an Officers'
Certificate.

            SECTION 2.06. Events of Default, Waiver. The Holders of a Majority
in Liquidation Amount of the Securities may, by vote, on behalf of the Holders,
waive any past Event of Default and its consequences. Upon such waiver, any such
Event of Default shall cease to exist, and any such Event of Default shall be
deemed to have been cured, for every


                                      -5-
<PAGE>

purpose of this Guarantee Agreement, but no such waiver shall extend to any
subsequent or other default or Event of Default or impair any right consequent
therefrom.

            SECTION 2.07. Event of Default; Notice. (a) The Guarantee Trustee
shall, within 90 days after the occurrence of an Event of Default known to a
Responsible Officer of the Trustee, transmit by mail, first class postage
prepaid, to the Holders, notices of all Events of Default known to the Guarantee
Trustee, unless such Events of Default have been cured before the giving of such
notice; provided, that, except in the case of a default in the payment of a
Guarantee Payment, the Guarantee Trustee shall be protected in withholding such
notice if and so long as the Board of Directors, the executive committee or a
trust committee of directors and/or a Responsible Officer in good faith
determines that the withholding of such notice is in the interests of the
Holders.

            (b) The Guarantee Trustee shall not be deemed to have knowledge of
any Event of Default unless a Responsible Officer charged with the
administration of the Declaration of Trust shall have received written notice of
such Event of Default.

            SECTION 2.08. Conflicting Interests. The Declaration of Trust shall
be deemed to be specifically described in this Guarantee Agreement for the
purposes of clause (i) of the first proviso contained in Section 310(b) of the
Trust Indenture Act.

                                   ARTICLE III

                        Powers, Duties and Rights of the
                                Guarantee Trustee

            SECTION 3.01. Powers and Duties of the Guarantee Trustee. (a) This
Guarantee Agreement shall be held by the Guarantee Trustee for the benefit of
the Holders, and the Guarantee Trustee shall not transfer this Guarantee
Agreement to any Person except a Holder exercising his or her rights pursuant to
Section 5.04(iv) or to a Successor Guarantee Trustee on acceptance by such
Successor Guarantee Trustee of its appointment to act as Successor Guarantee
Trustee. The right, title and interest of the Guarantee Trustee shall
automatically vest in any Successor Guarantee Trustee, upon acceptance by such
Successor Guarantee Trustee of its appointment hereunder, and such vesting and
cessation of title shall be effective whether or not conveyancing documents have
been executed and delivered pursuant to the appointment of such Successor
Guarantee Trustee.

            (b) If an Event of Default has occurred and is continuing, the
Guarantee Trustee shall enforce this Guarantee Agreement for the benefit of the
Holders.

            (c) The Guarantee Trustee, before the occurrence of any Event of
Default and after the curing of all Events of Default that may have occurred,
shall undertake to perform only such duties as are specifically set forth in
this Guarantee Agreement, and no implied covenants shall be read into this
Guarantee Agreement against the Guarantee Trustee. In case an Event of Default
has occurred (that has not been cured or waived pursuant to


                                      -6-
<PAGE>

Section 2.06), the Guarantee Trustee shall exercise such of the rights and
powers vested in it by this Guarantee Agreement, and use the same degree of care
and skill in its exercise thereof as a prudent person would exercise or use
under the circumstances in the conduct of his or her own affairs.

            (d) No provision of this Guarantee Agreement shall be construed to
relieve the Guarantee Trustee from liability for its own negligent action, its
own negligent failure to act or its own willful misconduct, except that:

            (i) prior to the occurrence of any Event of Default and after the
      curing or waiving of all such Events of Default that may have occurred:

                 (A) the duties and obligations of the Guarantee Trustee shall
            be determined solely by the express provisions of this Guarantee
            Agreement, and the Guarantee Trustee shall not be liable except for
            the performance of such duties and obligations as are specifically
            set forth in this Guarantee Agreement; and

                 (B) in the absence of bad faith on the part of the Guarantee
            Trustee, the Guarantee Trustee may conclusively rely, as to the
            truth of the statements and the correctness of the opinions
            expressed therein, upon any certificates or opinions furnished to
            the Guarantee Trustee and conforming to the requirements of this
            Guarantee Agreement; but in the case of any such certificates or
            opinions that by any provision hereof or of the Trust Indenture Act
            are specifically required to be furnished to the Guarantee Trustee,
            the Guarantee Trustee shall be under a duty to examine the same to
            determine whether or not they conform to the requirements of this
            Guarantee Agreement;

            (ii) the Guarantee Trustee shall not be liable for any error of
      judgment made in good faith by a Responsible Officer of the Guarantee
      Trustee, unless it shall be proved that the Guarantee Trustee was
      negligent in ascertaining the pertinent facts upon which such judgment was
      made;

            (iii) the Guarantee Trustee shall not be liable with respect to any
      action taken or omitted to be taken by it in good faith in accordance with
      the direction of the Holders of not less than a Majority in Liquidation
      Amount of the Securities relating to the time, method and place of
      conducting any proceeding for any remedy available to the Guarantee
      Trustee, or exercising any trust or power conferred upon the Guarantee
      Trustee under this Guarantee Agreement; and

            (iv) no provision of this Guarantee Agreement shall require the
      Guarantee Trustee to expend or risk its own funds or otherwise incur
      personal financial liability in the performance of any of its duties or in
      the exercise of any of its rights or powers if the Guarantee Trustee shall
      have reasonable grounds for believing that the repayment of such funds or
      liability is not assured to it under the terms of this Guarantee Agreement


                                      -7-
<PAGE>

      or indemnity satisfactory to it against such risk or liability is not
      reasonably assured to it.

            SECTION 3.02. Certain Rights of Guarantee Trustee. (a) Subject to
the provisions of Section 3.01:

            (i) The Guarantee Trustee may conclusively rely and shall be fully
      protected in acting or refraining from acting upon any resolution,
      certificate, statement, instrument, opinion, report, notice, request,
      direction, consent, order, bond, debenture, note, other evidence of
      indebtedness or other paper or document reasonably believed by it to be
      genuine and to have been signed, sent or presented by the proper party or
      parties.

            (ii) Any direction or act of the Guarantor contemplated by this
      Guarantee Agreement shall be sufficiently evidenced by all Officers'
      Certificate unless otherwise prescribed herein.

            (iii) Whenever, in the administration of this Guarantee Agreement,
      the Guarantee Trustee shall deem it desirable that a matter be proved or
      established before taking, suffering or omitting to take any action
      hereunder, the Guarantee Trustee (unless other evidence is herein
      specifically prescribed) may, in the absence of bad faith on its part,
      request and conclusively rely upon an Officers' Certificate which, upon
      receipt of such request from the Guarantee Trustee, shall be promptly
      delivered by the Guarantor.

            (iv) The Guarantee Trustee may consult with legal counsel, and the
      advice or written opinion of such legal counsel with respect to legal
      matters shall be full and complete authorization and protection in respect
      of any action taken, suffered or omitted to be taken by it hereunder in
      good faith and in accordance with such advice or opinion. Such legal
      counsel may be legal counsel to the Guarantor or any of its Affiliates and
      may be one of its employees. The Guarantee Trustee shall have the right at
      any time to seek instructions concerning the administration of this
      Guarantee Agreement from any court of competent jurisdiction.

            (v) The Guarantee Trustee shall be under no obligation to exercise
      any of the rights or powers vested in it by this Guarantee Agreement at
      the request or direction of any Holder, unless such Holder shall have
      provided to the Guarantee Trustee such security and indemnity reasonably
      satisfactory to it, against the costs, expenses (including attorneys' fees
      and expenses) and liabilities that might be incurred by it in complying
      with such request or direction, including such reasonable advances as may
      be requested by the Guarantee Trustee; provided, that nothing contained in
      this Section 3.02(a)(v) shall be taken to relieve the Guarantee Trustee,
      upon the occurrence of an Event of Default, of its obligation to exercise
      the rights and powers vested in it by this Guarantee Agreement.

            (vi) The Guarantee Trustee shall not be bound to make any
      investigation into the facts or matters stated in any resolution,
      certificate, statement, instrument, opinion,


                                      -8-
<PAGE>

      report, notice, request, direction, consent, order, bond, debenture, note,
      other evidence of indebtedness or other paper or document, but the
      Guarantee Trustee, in its discretion, may make such further inquiry or
      investigation into such facts or matters as it may see fit.

            (vii) The Guarantee Trustee may execute any of the trusts or powers
      hereunder or perform any duties hereunder either directly or by or through
      its agents or attorneys, and the Guarantee Trustee shall not be
      responsible for any misconduct or negligence on the part of any such agent
      or attorney appointed with due care by it hereunder.

            (viii) Whenever in the administration of this Guarantee Agreement
      the Guarantee Trustee shall deem it desirable to receive instructions with
      respect to enforcing any remedy or right or taking any other action
      hereunder, the Guarantee Trustee (A) may request instructions from the
      Holders, (B) may refrain from enforcing such remedy or right or taking
      such other action until such instructions are received and (C) shall be
      fully protected in acting in accordance with such instructions.

            (b) No provision of this Guarantee Agreement shall be deemed to
impose any duty or obligation on the Guarantee Trustee to perform any act or
acts or exercise any right, power, duty or obligation conferred or imposed on it
in any jurisdiction in which it shall be illegal, or in which the Guarantee
Trustee shall be unqualified or incompetent in accordance with applicable law,
to perform any such act or acts or to exercise any such right, power, duty or
obligation. No permissive power or authority available to the Guarantee Trustee
shall be construed to be a duty to act in accordance with such power and
authority.

            SECTION 3.03. Indemnity. The Guarantor agrees to indemnify the
Guarantee Trustee and its directors, officers, agents and employees for, and to
hold them harmless against, any loss, liability or expense incurred without
negligence or bad faith on the part of the Guarantee Trustee, arising out of or
in connection with the acceptance or administration of this Guarantee Agreement,
including the costs and expenses of defending itself against any claim or
liability in connection with the exercise or performance of any of its powers or
duties hereunder. The Guarantee Trustee will not claim or exact any lien or
charge on any Guarantee Payments as a result of any amount due to it under this
Guarantee Agreement. This indemnity shall survive the termination of this
Guarantee Agreement or the resignation or removal of the Guarantee Trustee.

            SECTION 3.04. Expenses. The Guarantor, as obligor on the Junior
Subordinated Debt Securities, shall from time to time reimburse the Guarantee
Trustee for its reasonable expenses and costs incurred in connection with the
performance of its duties hereunder. The provisions of this Section 3.04 shall
survive the termination of this Guarantee Agreement or the resignation or
removal of the Guarantee Trustee.


                                      -9-
<PAGE>

                                   ARTICLE IV

                                Guarantee Trustee

            SECTION 4.01. Guarantee Trustee; Eligibility. (a) There shall at all
times be a Guarantee Trustee which shall:

            (i) not be an Affiliate of the Guarantor; and

            (ii) be a Person that is eligible pursuant to the Trust Indenture
      Act to act as such and has a combined capital and surplus of at least
      $50,000,000, and shall be a corporation meeting the requirements of
      Section 310(c) of the Trust Indenture Act. If such corporation publishes
      reports of condition at least annually, pursuant to law or to the
      requirements of the supervising or examining authority, then, for the
      purposes of this Section and to the extent permitted by the Trust
      Indenture Act, the combined capital and surplus of such corporation shall
      be deemed to be its combined capital and surplus as set forth in its most
      recent report of condition so published.

            (b) If at any time the Guarantee Trustee shall cease to be eligible
to so act under Section 4.01(a), the Guarantee Trustee shall immediately resign
in the manner and with the effect set out in Section 4.02(c).

            (c) If the Guarantee Trustee has or shall acquire any "conflicting
interest" within the meaning of Section 310(b) of the Trust Indenture Act, the
Guarantee Trustee and Guarantor shall in all respects comply with the provisions
of Section 310(b) of the Trust Indenture Act.

            SECTION 4.02. Appointment, Removal and Resignation of the Guarantee
Trustee. (a) Subject to Section 4.02(b), in the absence of the existence of an
Event of Default, the Guarantee Trustee may be appointed or removed without
cause at any time by the Guarantor.

            (b) The Guarantee Trustee shall not be removed until a Successor
Guarantee Trustee has been appointed and has accepted such appointment by
written instrument executed by such Successor Guarantee Trustee and delivered to
the Guarantor.

            (c) The Guarantee Trustee appointed hereunder shall hold office
until a Successor Guarantee Trustee shall have been appointed or until its
removal or resignation. The Guarantee Trustee may resign from office (without
need for prior or subsequent accounting) by an instrument in writing executed by
the Guarantee Trustee and delivered to the Guarantor, which resignation shall
not take effect until a Successor Guarantee Trustee has been appointed and has
accepted such appointment by instrument in writing executed by such Successor
Guarantee Trustee and delivered to the Guarantor and the resigning Guarantee
Trustee.


                                      -10-
<PAGE>

            (d) If no Successor Guarantee Trustee shall have been appointed and
accepted appointment as provided in this Section 4.02 within 60 days after
delivery to the Guarantor of an instrument of resignation, the resigning
Guarantee Trustee may petition, at the expense of the Guarantor, any court of
competent jurisdiction for appointment of a Successor Guarantee Trustee. Such
court may thereupon, after prescribing such notice, if any, as it may deem
proper, appoint a Successor Guarantee Trustee.

                                    ARTICLE V

                                    Guarantee

            SECTION 5.01. Guarantee. The Guarantor irrevocably and
unconditionally agrees to pay in full to the Holders the Guarantee Payments
(without duplication of amounts theretofore paid by or on behalf of the Issuer),
as and when due, regardless of any defense, right of set-off or counterclaim
which the Issuer may have or assert. The Guarantor's obligation to make a
Guarantee Payment may be satisfied by direct payment of the required amounts by
the Guarantor to the Holders or by causing the Issuer to pay such amounts to the
Holders. The Guarantor shall give prompt written notice to the Guarantee Trustee
in the event it makes any direct payment hereunder.

            SECTION 5.02 Waiver of Notice and Demand. The Guarantor hereby
waives notice of acceptance of the Guarantee Agreement and of any liability to
which it applies or may apply, presentment, demand for payment, any right to
require a proceeding first against the Guarantee Trustee, Issuer or any other
Person before proceeding against the Guarantor, protest, notice of nonpayment,
notice of dishonor, notice of redemption and all other notices and demands.

            SECTION 5.03.     Obligations  Not  Affected.   The   obligations,
covenants,  agreements  and  duties  of the  Guarantor  under  this  Guarantee
Agreement  shall in no way be affected or impaired by reason of the  happening
from time to time of any of the following:

            (a) the release or waiver, by operation of law or otherwise, of the
      performance or observance by the Issuer of any express or implied
      agreement, covenant, term or condition relating to the Trust Securities to
      be performed or observed by the Issuer;

            (b) the extension of time for the payment by the Issuer of all or
      any portion of the Distributions (other than any extension of time for
      payment of Distributions that results from the extension of any interest
      payment period on the Junior Subordinated Debt Securities as so provided
      in the Indenture), Redemption Price, Liquidation Distribution or any other
      sums payable under the terms of the Trust Securities or the extension of
      time for the performance of any other obligation under, arising out of, or
      in connection with, the Trust Securities;


                                      -11-
<PAGE>

            (c) any failure, omission, delay or lack of diligence on the part of
      the Holders to enforce, assert or exercise any right, privilege, power or
      remedy conferred on the Holders pursuant to the terms of the Trust
      Securities, or any action on the part of the Issuer granting indulgence or
      extension of any kind;

            (d) the voluntary or involuntary liquidation, dissolution, sale of
      any collateral, receivership, insolvency, bankruptcy, assignment for the
      benefit of creditors, reorganization, arrangement, composition or
      readjustment of debt of, or other similar proceedings affecting, the
      Issuer or any of the assets of the Issuer;

            (e) any invalidity of, or defect or deficiency in, the Trust
      Securities;

            (f) the settlement or compromise of any obligation guaranteed hereby
      or hereby incurred;

            (g) the consummation of the Exchange Offer; or

            (h) any other circumstance whatsoever that might otherwise
      constitute a legal or equitable discharge or defense of a guarantor, it
      being the intent of this Section 5.03 that the obligations of the
      Guarantor hereunder shall be absolute and unconditional under any and all
      circumstances.

            There shall be no obligation of the Holders to give notice to, or
obtain the consent of, the Guarantor with respect to the happening of any of the
foregoing.

            SECTION 5.04. Rights of Holders. The Guarantor expressly
acknowledges that: (i) this Guarantee Agreement will be deposited with the
Guarantee Trustee to be held for the benefit of the Holders; (ii) the Guarantee
Trustee has the right to enforce this Guarantee Agreement on behalf of the
Holders; (iii) subject to Section 3.02(v), the Holders of a Majority in
Liquidation Amount of the Securities have the right to direct the time, method
and place of conducting any proceeding for any remedy available to the Guarantee
Trustee in respect of this Guarantee Agreement or exercising any trust or power
conferred upon the Guarantee Trustee under this Guarantee Agreement; and (iv)
any Holder may institute a legal proceeding directly against the Guarantor to
enforce its rights under this Guarantee Agreement, without first instituting a
legal proceeding against the Issuer or any other Person.

            SECTION 5.05. Guarantee of Payment. This Guarantee Agreement creates
a guarantee of payment and not of collection. This Guarantee Agreement will not
be discharged except by payment of the Guarantee Payments in full (without
duplication of amounts theretofore paid by the Issuer) or upon distribution of
Junior Subordinated Debt Securities to Holders as provided in the Declaration of
Trust.

            SECTION 5.06 Subrogation. The Guarantor shall be subrogated to all
(if any ) rights of the Holders against the Issuer in respect of any amounts
paid to the Holders by the Guarantor under this Guarantee Agreement and shall
have the right to waive payment by


                                      -12-
<PAGE>

the Issuer pursuant to Section 5.01; provided, however, that the Guarantor shall
not (except to the extent required by mandatory provisions of law) be entitled
to enforce or exercise any rights which it may acquire by way of subrogation or
any indemnity, reimbursement or other agreement, in all cases as a result of
payment under this Guarantee Agreement, if at the time of any such payment, any
amounts are due and unpaid under this Guarantee Agreement. If any amount shall
be paid to the Guarantor in violation of the preceding sentence, the Guarantor
agrees to hold such amount in trust for the Holders and to pay over such amount
to the Holders.

            SECTION 5.07 Independent Obligations. The Guarantor acknowledges
that its obligations hereunder are independent of the obligations of the Issuer
with respect to the Trust Securities and that the Guarantor shall be liable as
principal and as debtor hereunder to make Guarantee Payments pursuant to the
terms of this Guarantee Agreement notwithstanding the occurrence of any event
referred to in subsections (a) through (g), inclusive, of Section 5.03 hereof.

                                   ARTICLE VI

                              Covenants and Ranking

            SECTION 6.01. Ranking.. This Guarantee Agreement will constitute an
unsecured obligation of the Guarantor and will rank (a) subordinate and junior
in right of payment to all Senior Debt of the Guarantor, to the same extent and
in the same manner that the Junior Subordinated Debt Securities are subordinated
to Senior Debt pursuant to the Indenture, and (b) senior to all capital stock of
the Guarantor.

            SECTION 6.02. Pari Passu Guarantees. This Guarantee Agreement shall
rank pari passu with any similar guarantee agreements issued by the Guarantor on
behalf of the holders of trust securities issued by a trust created by the
Guarantor similar to the Issuer.

            SECTION 6.03. Limitation on Transactions. The Guarantor covenants
and agrees with each Holder that it will not (i) declare or pay any dividends or
distributions on, or redeem, purchase, acquire or make a liquidation payment
with respect to, any shares of the Guarantor's capital stock (which includes
common and preferred stock), or (ii) make any payment of principal, interest or
premium, if any, on or repay, repurchase or redeem any debt securities of the
Guarantor (including Other Debentures) that rank pari passu with or junior in
interest to the Securities or (iii) make any guarantee payments with respect to
any guarantee by the Guarantor of debt securities of any subsidiary of the
Guarantor (including Other Guarantees) if such guarantee ranks pari passu with
or junior in interest to the Securities (other than (a) dividends or
distributions in common stock of the Guarantor, (b) any declaration of a
dividend in connection with the implementation of a stockholders' rights plan,
or the issuance of stock under any such plan in the future, or the redemption or
repurchase of any such rights pursuant thereto, (c) payments under this
Guarantee, (d) purchases or acquisitions of shares of the Guarantor's common
stock in connection with the satisfaction by the Guarantor of its obligations
under any employee benefit plan or other contractual obligation of the Guarantor


                                      -13-
<PAGE>

(other than a contractual obligation ranking pari passu with or junior in
interest to the Securities), (e) as a result of a reclassification of the
Guarantor's capital stock or the exchange or conversion of one class or series
of the Guarantor's capital stock for another class or series of the Guarantor's
capital stock, or (f) the purchase of fractional interests in shares of the
Guarantor's capital stock pursuant to the conversion or exchange provisions of
such capital stock or the security being converted or exchanged), if at such
time (i) there shall have occurred an Event of Default (as defined in the
Indenture), (ii) the Guarantor shall be in default with respect to its payment
of any obligations under this Guarantee or (iii) the Guarantor shall have given
notice of its election to begin an Extension Period (as defined in the
Indenture) and shall not have rescinded such notice, or such Extension Period,
or any extension thereof, shall be continuing.

            The Guarantor also covenants with each Holder for so long as the
Securities remain outstanding (i) to maintain 100% direct or indirect ownership
of the Common Securities; provided, however, that any permitted successor of the
Guarantor under the Indenture may succeed to the Guarantor's ownership of such
Common Securities, (ii) not to cause or permit the dissolution, winding-up or
termination of the Issuer, except (a) in connection with a distribution of the
Junior Subordinated Debt Securities to the holders of Capital Securities or (b)
in connection with certain mergers, consolidations or amalgamations permitted by
the Declaration of Trust and (iii) to use its reasonable efforts, consistent
with the terms and provisions of such Declaration of Trust, to cause the Trust
to remain classified as a grantor trust and not an association taxable as a
corporation for United States Federal income tax purposes.

            SECTION 6.04. Exchange Offer . In the event an Exchange Offer
Registration Statement (as defined in the Registration Rights Agreement) becomes
efective and the Issuer issues any Exchange Capital Securities (as defined in
the Registration Rights Agreement) in the Exchange Offer, the Guarantor will
enter into a new guarantee agreement, in substantially the same form as this
Guarantee, with respect to the Exchange Capital Securities.


                                      -14-
<PAGE>

                                   ARTICLE VII

                                   Termination

            SECTION 7.01. Termination. This Guarantee Agreement shall terminate
and be of no further force and effect upon (i) full payment of the Redemption
Price of all Trust Securities, (ii) the distribution of Junior Subordinated Debt
Securities to the Holders in exchange for all of the Trust Securities, (iii)
full payment of the amounts payable in accordance with the Declaration of Trust
upon liquidation of the Issuer or (iv) the exchange of all the Capital
Securities for Exchange Capital Securities. Notwithstanding the foregoing, this
Guarantee Agreement will continue to be effective or will be reinstated, as the
case may be, if at any time any Holder must repay any sums paid with respect to
Trust Securities or this Guarantee Agreement. The provisions of Sections 3.03
and 3.04 shall survive termination of this Guarantee Agreement as provided
therein.

                                  ARTICLE VIII

                                  Miscellaneous

            SECTION 8.01. Successors and Assigns. All guarantees and agreements
contained in this Guarantee Agreement shall bind the successors, assigns,
receivers, trustees and representatives of the Guarantor and shall inure to the
benefit of the Holders then outstanding. Except in connection with a
consolidation, merger or sale involving the Guarantor that is permitted under
Article VIII of the Indenture and pursuant to which the assignee agrees in
writing to perform the Guarantor's obligations hereunder, the Guarantor shall
not assign its obligations hereunder.

            SECTION 8.02. Amendments. Except with respect to any changes which
do not adversely affect the rights of the Holders in any material respect (in
which case no consent of the Holders will be required), this Guarantee Agreement
may only be amended with the prior approval of the Holders of not less than a
Majority in Liquidation Amount of the Securities. The provisions of Article VI
of the Declaration of Trust concerning meetings of the Holders shall apply to
the giving of such approval.

            SECTION 8.03. Notices. Any notice, request or other communication
required or permitted to be given hereunder shall be in writing, duly signed by
the party giving such notice, and delivered, telecopied (confirmed by delivery
of the original) or mailed by first class mail as follows:

            (a) if given to the Guarantor, to the address set forth below or
      such other address, facsimile number or to the attention of such other
      Person as the Guarantor may give notice to the Holders:


                                      -15-
<PAGE>

                  U.S.B. HOLDING CO., INC.
                  100 Dutch Hill Road
                  Orangeburg, New York  10962
                  Facsimile No.: 914-365-4695
                  Attention: Steven T. Sabatini

            (b) if given to the Issuer, in care of the Guarantee Trustee, at the
      Issuer's (and the Guarantee Trustee's) address set forth below or such
      other address as the Guarantee Trustee on behalf of the Issuer may give
      notice to the Holders:

                  UNION STATE CAPITAL TRUST I
                  c/o U.S.B. HOLDING CO., INC.
                  100 Dutch Hill Road
                  Orangeburg, New York  10962
                  Facsimile No.: 914-365-4695
                  Attention: Steven T. Sabatini

                  with a copy to:

                  The Chase Manhattan Bank
                  450 West 33rd Street, 15th Floor
                  New York, New York 10001
                  Facsimile No.:
                  Attention: Corporate Trustee Administration Department

            (c) if given to the Guarantee Trustee:

                  The Chase Manhattan Bank
                  450 West 33rd Street, 15th Floor
                  New York, New York 10001
                  Facsimile No.:
                  Attention: Corporate Trustee Administration Department

            (d) if given to any Holder, at the address set forth on the books
      and records of the Issuer.

            All notices hereunder shall be deemed to have been given when
received in person, telecopied with receipt confirmed, or mailed by first class
mail, postage prepaid, except that if a notice or other document is refused
delivery or cannot be delivered because of a changed address of which no notice
was given, such notice or other document shall be deemed to have been delivered
on the date of such refusal or inability to deliver.

            SECTION 8.04. Benefit. This Guarantee Agreement is solely for the
benefit of the Holders and is not separately transferable from the Trust
Securities.


                                      -16-
<PAGE>

            SECTION 8.05. Interpretation. In this Guarantee Agreement, unless
the context otherwise requires:

            (a) capitalized terms used in this Guarantee Agreement but not
      defined in the preamble hereto have the respective meanings assigned to
      them in Section 1.01;

            (b) a term defined anywhere in this Guarantee Agreement has the same
      meaning throughout;

            (c) all references to "the Guarantee Agreement" or "this Guarantee
      Agreement" are to this Guarantee Agreement as modified, supplemented or
      amended from time to time;

            (d) all references in this Guarantee Agreement to Articles and
      Sections are to Articles and Sections of this Guarantee Agreement unless
      otherwise specified;

            (e) a term defined in the Trust Indenture Act has the same meaning
      when used in this Guarantee Agreement unless otherwise defined in this
      Guarantee Agreement or unless the context otherwise requires;

            (f) a reference to the singular includes the plural and vice versa;
      and

            (g) the masculine, feminine or neuter genders used herein shall
      include the masculine, feminine and neuter genders.

            SECTION 8.06. Governing Law. THIS GUARANTEE AGREEMENT SHALL BE
GOVERNED BY AND CONSTRUED AND INTERPRETED IN ACCORDANCE WITH THE LAWS OF THE
STATE OF NEW YORK WITHOUT REGARD TO THE CONFLICT OF LAW PRINCIPLES THEREOF.

            This instrument may be executed in any number of counterparts, each
of which so executed shall be deemed to be an original, but all such
counterparts shall together constitute but one and the same instrument.


                                      -17-
<PAGE>

            THIS GUARANTEE AGREEMENT is executed as of the day and year first
above written.

                                       U.S.B. HOLDING CO., INC.


                                       By  /s/ [ILLEGIBLE]
                                           ----------------------------------
                                           Name:
                                           Title:


                                       THE CHASE MANHATTAN BANK


                                       By  /s/ Sheik Wiltshire
                                           ----------------------------------
                                           Name:  Sheik Wiltshire
                                           Title: (Second Vice President)