<PAGE>

                                                                     EXHIBIT 4.3



                              GUARANTEE AGREEMENT


                                    Between


                       PROVIDENT BANKSHARES CORPORATION
                                (as Guarantor)


                                      and


                             BANKERS TRUST COMPANY
                            (as Guarantee Trustee)


                                  dated as of


                             ______________, 2000
<PAGE>

                              PROVIDENT TRUST II
           Certain Sections of this Guarantee Agreement relating to
                        Sections 310 through 318 of the
                         Trust Indenture Act of 1939:


<TABLE>
<CAPTION>

Trust Indenture                                             Guarantee Agreement
 Act Section                                                     Section
-------------                                               -------------------
<S>            <C>                                               <C>
Section 310    (a) (1).........................................   4.1 (a)
               (a) (2).........................................   4.1 (a)
               (a) (3).........................................   Not Applicable
               (a) (4).........................................   Not Applicable
               (b).............................................   2.8, 4.1 (c)
Section 311    (a).............................................   Not Applicable
               (b).............................................   Not Applicable
Section 312    (a).............................................   2.2 (a)
               (b).............................................   2.2 (b)
               (c).............................................   Not Applicable
Section 313    (a).............................................   2.3
               (a) (4).........................................   2.3
               (b).............................................   2.3
               (c).............................................   2.3
               (d).............................................   2.3
Section 314    (a).............................................   2.4
               (b).............................................   2.4
               (c) (1).........................................   2.5
               (c) (2).........................................   2.5
               (c) (3).........................................   2.5
               (e).............................................   1.1, 2.5, 3.2
Section 315    (a).............................................   3.1 (d)
               (b).............................................   2.7
               (c).............................................   3.1 (c)
               (d).............................................   3.1 (d)
               (e).............................................   Not Applicable
Section 316    (a).............................................   1.1, 2.6, 5.4
               (a) (1) (A).....................................   5.4
               (a) (1) (B).....................................   5.4
               (a) (2).........................................   Not Applicable
               (b).............................................   5.3
               (c).............................................   Not Applicable
Section 317    (a) (1).........................................   Not Applicable
               (a) (2).........................................   Not Applicable
               (b).............................................   Not Applicable
Section 318    (a).............................................   2.1
</TABLE>

Note:  This reconciliation and tie shall not, for any purpose, be deemed to be a
       part of the Guarantee Agreement.
<PAGE>

                               TABLE OF CONTENTS
                               -----------------

<TABLE>
<CAPTION>
                                                           Page
                                                           ----

<S>               <C>                                     <C>
ARTICLE I.         DEFINITIONS

     Section 1.1.    Definitions.........................     2

ARTICLE II.        TRUST INDENTURE ACT
     Section 2.1.    Trust Indenture Act; Application....     5
     Section 2.2.    List of Holders.....................     5
     Section 2.3.    Reports by the Guarantee Trustee....     6
     Section 2.4.    Periodic Reports to Guarantee
                     Trustee.............................     6
     Section 2.5.    Evidence of Compliance with
                     Conditions Precedent................     6
     Section 2.6.    Events of Default; Waiver...........     6
     Section 2.7.    Event of Default; Notice............     6
     Section 2.8.    Conflicting Interests...............     7

ARTICLE III.       POWERS, DUTIES AND RIGHTS OF THE GUARANTEE
                   TRUSTEE
     Section 3.1.    Powers and Duties of the Guarantee
                     Trustee.............................     7
     Section 3.2.    Certain Rights of Guarantee Trustee.     9
     Section 3.3.    Indemnity...........................    10
     Section 3.4.    Expenses............................    11

ARTICLE IV.        GUARANTEE TRUSTEE
     Section 4.1.    Guarantee Trustee; Eligibility......    11
     Section 4.2.    Appointment, Removal and Resignation
                     of the Guarantee Trustee............    11

ARTICLE V.         GUARANTEE
     Section 5.1.    Guarantee...........................    12
     Section 5.2.    Waiver of Notice and Demand.........    12
     Section 5.3.    Obligations Not Affected............    13
     Section 5.4.    Rights of Holders...................    13
     Section 5.5.    Guarantee of Payment................    14
     Section 5.6.    Subrogation.........................    14
     Section 5.7.    Independent Obligations.............    14

ARTICLE VI.        COVENANTS AND SUBORDINATION
     Section 6.1.    Subordination.......................    14
     Section 6.2.    Pari Passu Guarantees...............    15
</TABLE>
<PAGE>

<TABLE>
<CAPTION>
                                                           Page
                                                           ----

<S>               <C>                                     <C>
ARTICLE VII.       TERMINATION
     Section 7.1     Termination.........................    15

ARTICLE VIII.      MISCELLANEOUS
     Section 8.1.    Successors and Assigns..............    15
     Section 8.2.    Amendments..........................    16
     Section 8.3.    Notices.............................    16
     Section 8.4.    Benefit.............................    17
     Section 8.5.    Interpretation......................    17
     Section 8.6.    Governing Law.......................    18
     Section 8.7.    Counterparts........................    18
</TABLE>
<PAGE>


                              GUARANTEE AGREEMENT
                              -------------------


  THIS GUARANTEE AGREEMENT, dated as of ______________, 2000 (this "Guarantee
Agreement"), is executed and delivered by PROVIDENT BANKSHARES  CORPORATION,  a
Maryland corporation (the "Guarantor"), having its principal office at 114 East
Lexington Street, Baltimore, Maryland 21202 and BANKERS TRUST COMPANY, a New
York banking corporation, having its principal office at Four Albany Street,
Fourth Floor, New York, New York 10006, as trustee, for the benefit of the
Holders (as defined herein) from time to time of the Preferred Securities (as
defined herein) of Provident Trust II, a Delaware statutory business trust (the
"Issuer Trust").

                                   RECITALS

  WHEREAS, pursuant to an Amended and Restated Trust Agreement (the "Trust
Agreement"), dated as of ______________, 2000, among Provident Bankshares
Corporation,  as Depositor, Bankers Trust Company, as Property Trustee (the
"Property Trustee"), Bankers Trust (Delaware), as Delaware Trustee (the
"Delaware Trustee") (collectively, the "Issuer Trustees") and the Holders from
time to time of preferred undivided beneficial ownership interests in the assets
of the Issuer Trust, the Issuer Trust is issuing up to $___________ aggregate
Liquidation Amount (as defined herein) of its _______% Preferred Securities,
Liquidation Amount $_______ per preferred security (the "Preferred Securities"),
representing preferred undivided beneficial ownership interests in the assets of
the Issuer Trust and having the terms set forth in the Trust Agreement;

  WHEREAS, the Preferred Securities will be issued by the Issuer Trust and the
proceeds thereof, together with the proceeds from the issuance of the Issuer
Trust's Common Securities (as defined herein), will be used to purchase the
Junior Subordinated Debentures due ___________, 2030 (as defined in the Trust
Agreement) (the "Junior Subordinated Debentures") of the Guarantor which will be
deposited with Bankers Trust Company, as Property Trustee under the Trust
Agreement, as trust assets; and

  WHEREAS, as incentive for the Holders to purchase Preferred Securities, the
Guarantor desires irrevocably and unconditionally to agree, to the extent set
forth herein, to pay to the Holders of the Preferred Securities the Guarantee
Payments (as defined herein) and to make certain other payments on the terms and
conditions set forth herein.

  NOW, THEREFORE, in consideration of the purchase of the Preferred Securities
by each Holder, which purchase the Guarantor hereby acknowledges shall benefit
the Guarantor, and intending to be legally bound hereby, the Guarantor executes
and delivers this Guarantee Agreement for the benefit of the Holders from time
to time of the Preferred Securities.

                                       1
<PAGE>

                                   ARTICLE I


                                  DEFINITIONS

Section 1.1.  Definitions.

  As used in this Guarantee Agreement, the terms set forth below shall, unless
the context otherwise requires, have the following meanings.  Capitalized terms
used but not otherwise defined herein shall have the meanings assigned to such
terms in the Trust Agreement as in effect on the date hereof.

  "Additional Amount" has the meaning specified in the Trust Agreement.

  "Affiliate" of any specified Person means any other Person directly or
indirectly controlling or controlled by or under direct or indirect common
control with such specified Person.  For the purposes of this definition,
"control" when used with respect to any specified Person means the power to
direct the management and policies of such Person, directly or indirectly,
whether through the ownership of voting securities, by contract or otherwise;
and the terms "controlling" and "controlled" have meanings correlative to the
foregoing.

  "Common Securities" means the securities representing common undivided
beneficial interests in the assets of the Issuer Trust.

  "Delaware Trustee" shall have the meaning specified in the first recital of
this Guarantee Agreement.

  "Distributions" means preferential cumulative cash distributions accumulating
from ______________, 2000 and payable quarterly in arrears on March 31, June 30,
September 30, and December 31 of each year, commencing ___________, 2000 at the
annual rate of _______% of  the Liquidation Amount.

  "Event of Default" means (a) a default by the Guarantor in any of its payment
obligations under this Guarantee Agreement, or (b) a default by the Guarantor in
any other obligation hereunder that remains unremedied for 30 days.

  "Guarantee Agreement" means this Guarantee Agreement, as modified, amended or
supplemented from time to time.

  "Guarantee Payments" means the following payments or distributions, without
duplication, with respect to the Preferred Securities, to the extent not paid or
made by or on behalf of the Issuer Trust: (a) any accrued and unpaid
Distributions (as defined in the Trust Agreement) required to be paid on the
Preferred Securities, to the extent the Issuer Trust shall have funds on hand
available therefor at such time, (b) the Redemption Price, with respect to the
Preferred Securities called for redemption by the Issuer Trust to the extent
that the Issuer Trust shall have funds on hand available therefor at such time,
and (c) upon a voluntary or involuntary termination, winding-up or liquidation
of the Issuer Trust, unless the Junior Subordinated Debentures are distributed
to the Holders, the lesser of (i) the aggregate of the Liquidation Amount and
all accumulated and unpaid Distributions to the date of payment to the extent
the Issuer Trust shall

                                       2
<PAGE>

have funds on hand available to make such payment at such time and (ii) the
amount of assets of the Issuer Trust remaining available for distribution to
Holders in liquidation of the Issuer Trust (in either case, the "Liquidation
Distribution").

  "Guarantee Trustee" means Bankers Trust Company, until a Successor Guarantee
Trustee has been appointed and has accepted such appointment pursuant to the
terms of this Guarantee Agreement and thereafter means each such Successor
Guarantee Trustee.

  "Guarantor" shall have the meaning specified in the first paragraph of this
Guarantee Agreement.

  "Holder" means any holder, as registered on the books and records of the
Issuer Trust, of any Preferred Securities; provided, however, that, in
determining whether the holders of the requisite percentage of Preferred
Securities have given any request, notice, consent or waiver hereunder, "Holder"
shall not include the Guarantor, the Guarantee Trustee, or any Affiliate of the
Guarantor or the Guarantee Trustee.

  "Indenture" means the Junior Subordinated Indenture dated as of
______________, 2000, between Provident Bankshares Corporation and Bankers Trust
Company, as trustee, as may be modified, amended or supplemented from time to
time.

  "Issuer Trust" shall have the meaning specified in the first paragraph of this
Guarantee Agreement.

  "Issuer Trustees" shall have the meaning specified in the first recital of
this Guarantee Agreement.

  "Junior Subordinated Debentures" shall have the meaning specified in the first
recital of this Guarantee Agreement.

  "Like Amount" means (a) with respect to a redemption of Preferred Securities,
Preferred Securities having a Liquidation Amount equal to the principal amount
of Junior Subordinated Debentures to be contemporaneously redeemed in accordance
with the Indenture, the proceeds of which will be used to pay the Redemption
Price of such Preferred Securities, (b) with respect to a distribution of Junior
Subordinated Debentures to Holders of Preferred Securities in connection with a
dissolution or liquidation of the Issuer Trust, Junior Subordinated Debentures
having a principal amount equal to the Liquidation Amount of the Preferred
Securities of the Holder to whom such Junior Subordinated Debentures are
distributed, and (c) with respect to any distribution of an Additional Amount to
Holders of Preferred Securities, Junior Subordinated Debentures having a
principal amount equal to the Liquidation Amount of the Preferred Securities in
respect of which such distribution is made.

  "Liquidation Amount" means the stated amount of $_______ per Preferred
Security.

                                       3
<PAGE>

  "Majority in Liquidation Amount of the Preferred Securities" means, except as
provided by the Trust Indenture Act, Preferred Securities representing more than
50% of the aggregate Liquidation Amount of all then outstanding Preferred
Securities issued by the Issuer Trust.

  "Officers' Certificate" means, with respect to any Person, a certificate
signed by the Chairman of the Board, Chief Executive Officer, President or a
Vice President, and by the Chief Financial Officer, Treasurer, an Associate
Treasurer, an Assistant Treasurer, the Secretary or an Assistant Secretary of
such Person, and delivered to the Guarantee Trustee.  Any Officers' Certificate
delivered with respect to compliance with a condition or covenant provided for
in this Guarantee Agreement shall include:

     (a) a statement by each officer signing the Officers' Certificate that
such officer has read the covenant or condition and the definitions relating
thereto;

     (b) a brief statement of the nature and scope of the examination or
investigation undertaken by such officer in rendering the Officers' Certificate;

     (c) a statement that such officer has made such examination or
investigation as, in such officer's opinion, is necessary to enable such officer
to express an informed opinion as to whether or not such covenant or condition
has been complied with; and

     (d) a statement as to whether, in the opinion of such officer, such
condition or covenant has been complied with.

  "Person" means a legal person, including any individual, corporation, estate,
partnership, joint venture, association, joint stock company, limited liability
company, trust, unincorporated association, or government or any agency or
political subdivision thereof, or any other entity of whatever nature.

  "Preferred Securities" shall have the meaning specified in the first recital
of this Guarantee Agreement.

  "Property Trustee" shall have the meaning specified in the first recital of
this Guarantee Agreement.

  "Redemption Date" means, with respect to any Preferred Security to be
redeemed, the date fixed for such redemption by or pursuant to the Trust
Agreement; provided that each Junior Subordinated Debenture Redemption Date (as
such term is defined in the Indenture) and the stated maturity of the Junior
Subordinated Debentures shall be a Redemption Date for a Like Amount of
Preferred Securities.

  "Redemption Price" shall have the meaning specified in the Trust Agreement.

  "Responsible Officer" means, when used with respect to the Guarantee Trustee,
any officer assigned to the Corporate Trust Office, including any managing
director, principal, vice president, assistant vice president, assistant
treasurer, assistant secretary or any other officer of the Guarantee

                                       4
<PAGE>

Trustee customarily performing functions similar to those performed by any of
the above designated officers and having direct responsibility for the
administration of this Guarantee Agreement, and also, with respect to a
particular matter, any other officer to whom such matter is referred because of
such officer's knowledge of and familiarity with the particular subject.

  "Senior Indebtedness" shall have the meaning specified in the Indenture.

  "Successor Guarantee Trustee" means a successor Guarantee Trustee possessing
the qualifications to act as Guarantee Trustee under Section 4.1.

  "Trust Agreement" means the Amended and Restated Trust Agreement, dated
______________, 2000, executed by Provident Bankshares Corporation, as
depositor, Bankers Trust (Delaware), as Delaware Trustee, and Bankers Trust
Company, as Property Trustee.

  "Trust Indenture Act" means the Trust Indenture Act of 1939, as amended by the
Trust Indenture Reform Act of 1990, or any successor statute, in each case as
amended from time to time.

                                   ARTICLE II

                              TRUST INDENTURE ACT

Section 2.1.  Trust Indenture Act; Application.

  If any provision hereof limits, qualifies or conflicts with a provision of the
Trust Indenture Act that is required under such Act to be a part of and govern
this Guarantee Agreement, the provision of the Trust Indenture Act shall
control.  If any provision of this Guarantee Agreement modifies or excludes any
provision of the Trust Indenture Act that may be so modified or excluded, the
latter provision shall be deemed to apply to this Guarantee Agreement as so
modified or excluded, as the case may be.

Section 2.2.  List of Holders.

     (a) The Guarantor will furnish or cause to be furnished to the Guarantee
Trustee:

         (i) quarterly, not more than 15 days after March 15, June 15, September
     15 and December 15 in each year, a list, in such form as the Guarantee
     Trustee may reasonably require, of the names and addresses of the Holders
     as of such date; and

         (ii) at such other times as the Guarantee Trustee may request in
     writing, within 30 days after the receipt by the Guarantor of any such
     request, a list of similar form and content as of a date not more than 15
     days prior to the time such list is furnished.

     (b) The Guarantee Trustee shall comply with the requirements of Section
312(b) of the Trust Indenture Act.

                                       5
<PAGE>

Section 2.3.  Reports by the Guarantee Trustee.

  Within 60 days of January 31 of each year commencing January 31, 2001, the
Guarantee Trustee shall provide to the Holders such reports, if any, as are
required by Section 313 of the Trust Indenture Act in the form and in the manner
provided by Section 313 of the Trust Indenture Act.  The Guarantee Trustee shall
also comply with the requirements of Section 313(d) of the Trust Indenture Act.

Section 2.4.  Periodic Reports to the Guarantee Trustee.

  The Guarantor shall provide to the Guarantee Trustee and the Holders such
documents, reports and information, if any, as required by Section 314 of the
Trust Indenture Act and the compliance certificate required by Section 314 of
the Trust Indenture Act, in the form, in the manner and at the times required by
Section 314 of the Trust Indenture Act.

Section 2.5.  Evidence of Compliance with Conditions Precedent.

  The Guarantor shall provide to the Guarantee Trustee such evidence of
compliance with such conditions precedent, if any, provided for in this
Guarantee Agreement that relate to any of the matters set forth in Section
314(c) of the Trust Indenture Act.  Any certificate or opinion required to be
given by an officer pursuant to Section 314(c)(1) may be given in the form of an
Officers' Certificate.

Section 2.6.  Events of Default; Waiver.

  The Holders of a Majority in Liquidation Amount of the Preferred Securities
may, by vote, on behalf of the Holders, waive any past Event of Default and its
consequences.  Upon such waiver, any such Event of Default shall cease to exist,
and any Event of Default arising therefrom shall be deemed to have been cured,
for every purpose of this Guarantee Agreement, but no such waiver shall extend
to any subsequent or other default or Event of Default or impair any right
consequent therefrom.

Section 2.7.  Event of Default; Notice.

     (a) The Guarantee Trustee shall, within 90 days after the occurrence of an
Event of Default, transmit by mail, first class postage prepaid, to the Holders,
notices of all Events of Default known to the Guarantee Trustee, unless such
Events of Default have been cured before the giving of such notice; provided
that, except in the case of a default in the payment of a Guarantee Payment, the
Guarantee Trustee shall be protected in withholding such notice if and so long
as the Board of Directors, the executive committee or a trust committee of
directors and/or Responsible Officers of the Guarantee Trustee in good faith
determines that the withholding of such notice is in the interests of the
Holders.

     (b) The Guarantee Trustee shall not be deemed to have knowledge of any
Event of Default unless (i) a Responsible Officer charged with the
administration of this Guarantee Agreement shall have received written notice of
such Event of Default, or (ii) a Responsible Officer

                                       6
<PAGE>

of the Guarantee Trustee charged with administration of the Trust Agreement
shall have obtained actual knowledge thereof.

Section 2.8.  Conflicting Interests.

  The Trust Agreement shall be deemed to be specifically described in this
Guarantee Agreement for the purposes of clause (i) of the first proviso
contained in Section 310(b) of the Trust Indenture Act.

                                  ARTICLE III

                        POWERS, DUTIES AND RIGHTS OF THE
                               GUARANTEE TRUSTEE

Section 3.1.  Powers and Duties of the Guarantee Trustee.

     (a) This Guarantee Agreement shall be held by the Guarantee Trustee for the
benefit of the Holders, and the Guarantee Trustee shall not transfer this
Guarantee Agreement to any Person except to a Holder exercising his or her
rights pursuant to Section 5.4(d) or to a Successor Guarantee Trustee on
acceptance by such Successor Guarantee Trustee of its appointment to act as
Successor Guarantee Trustee hereunder. The right, title and interest of the
Guarantee Trustee, as such, hereunder shall automatically vest in any Successor
Guarantee Trustee, upon acceptance by such Successor Guarantee Trustee of its
appointment hereunder, and such vesting and cessation of title shall be
effective whether or not conveyancing documents have been executed and delivered
pursuant to the appointment of such Successor Guarantee Trustee.

     (b) If an Event of Default has occurred and is continuing, the Guarantee
Trustee shall enforce this Guarantee Agreement for the benefit of the Holders.

     (c) The Guarantee Trustee, before the occurrence of any Event of Default
and after the curing of all Events of Default that may have occurred, shall be
obligated to perform only such duties as are specifically set forth in this
Guarantee Agreement (including pursuant to Section 2.1), and no implied
covenants shall be read into this Guarantee Agreement against the Guarantee
Trustee. If an Event of Default has occurred (that has not been cured or waived
pursuant to Section 2.6), the Guarantee Trustee shall exercise such of the
rights and powers vested in it by this Guarantee Agreement, and use the same
degree of care and skill in its exercise thereof, as a prudent person would
exercise or use under the circumstances in the conduct of his or her own
affairs.

     (d) No provision of this Guarantee Agreement shall be construed to relieve
the Guarantee Trustee from liability for its own negligent action, its own
negligent failure to act or its own bad faith or willful misconduct, except
that:

         (i) prior to the occurrence of any Event of Default and after the
     curing or waiving of all such Events of Default that may have occurred:

                                       7
<PAGE>

               (A) the duties and obligations of the Guarantee Trustee shall be
         determined solely by the express provisions of this Guarantee Agreement
         (including pursuant to Section 2.1), and the Guarantee Trustee shall
         not be liable except for the performance of such duties and obligations
         as are specifically set forth in this Guarantee Agreement (including
         those set forth in Section 2.1); and

               (B) in the absence of bad faith on the part of the Guarantee
         Trustee, the Guarantee Trustee may conclusively rely, as to the truth
         of the statements and the correctness of the opinions expressed
         therein, upon any certificates or opinions furnished to the Guarantee
         Trustee and conforming to the requirements of this Guarantee Agreement;
         but in the case of any such certificates or opinions that by any
         provision hereof or of the Trust Indenture Act are specifically
         required to be furnished to the Guarantee Trustee, the Guarantee
         Trustee shall be under a duty to examine the same to determine whether
         or not they conform to the requirements of this Guarantee Agreement;

         (ii) the Guarantee Trustee shall not be liable for any error of
     judgment made in good faith by a Responsible Officer of the Guarantee
     Trustee, unless it shall be proved that the Guarantee Trustee was negligent
     in ascertaining the pertinent facts upon which such judgment was made;

         (iii) the Guarantee Trustee shall not be liable with respect to any
     action taken or omitted to be taken by it in good faith in accordance with
     the direction of the Holders of not less than a Majority in Liquidation
     Amount of the Preferred Securities relating to the time, method and place
     of conducting any proceeding for any remedy available to the Guarantee
     Trustee, or exercising any trust or power conferred upon the Guarantee
     Trustee under this Guarantee Agreement; and

         (iv) no provision of this Guarantee Agreement shall require the
     Guarantee Trustee to expend or risk its own funds or otherwise incur
     personal financial liability in the performance of any of its duties or in
     the exercise of any of its rights or powers if the Guarantee Trustee shall
     have reasonable grounds for believing that the repayment of such funds or
     liability is not assured to it under the terms of this Guarantee Agreement
     or adequate indemnity against such risk or liability is not reasonably
     assured to it.

Section 3.2.  Certain Rights of Guarantee Trustee.

     (a) Subject to the provisions of Section 3.1:

         (i) the Guarantee Trustee may conclusively rely and shall be fully
     protected in acting or refraining from acting upon any resolution,
     certificate, statement, instrument, opinion, report, notice, request,
     direction, consent, order, bond, debenture, note, other evidence of
     indebtedness or other paper or document reasonably believed by it to be
     genuine and to have been signed, sent or presented by the proper party or
     parties;

                                       8
<PAGE>

         (ii) any direction or act of the Guarantor contemplated by this
     Guarantee Agreement shall be sufficiently evidenced by an Officers'
     Certificate unless otherwise prescribed herein;

         (iii) whenever, in the administration of this Guarantee Agreement, the
     Guarantee Trustee shall deem it desirable that a matter be proved or
     established before taking, suffering or omitting to take any action
     hereunder, the Guarantee Trustee (unless other evidence is herein
     specifically prescribed) may, in the absence of bad faith on its part,
     request and conclusively rely upon an Officers' Certificate which, upon
     receipt of such request from the Guarantee Trustee, shall be promptly
     delivered by the Guarantor;

         (iv) the Guarantee Trustee may consult with legal counsel, and the
     advice or written opinion of such legal counsel with respect to legal
     matters shall be full and complete authorization and protection in respect
     of any action taken, suffered or omitted to be taken by it hereunder in
     good faith and in accordance with such advice or opinion. Such legal
     counsel may be legal counsel to the Guarantor or any of its Affiliates and
     may be one of its employees. The Guarantee Trustee shall have the right at
     any time to seek instructions concerning the administration of this
     Guarantee Agreement from any court of competent jurisdiction;

         (v) the Guarantee Trustee shall be under no obligation to exercise any
     of the rights or powers vested in it by this Guarantee Agreement at the
     request or direction of any Holder, unless such Holder shall have provided
     to the Guarantee Trustee such security and indemnity as would satisfy a
     reasonable person in the position of the Guarantee Trustee, against the
     costs, expenses (including attorneys' fees and expenses) and liabilities
     that might be incurred by it in complying with such request or direction,
     including such reasonable advances as may be requested by the Guarantee
     Trustee; provided, however, that nothing herein shall relieve the Guarantee
     Trustee of its obligations upon the occurrence of an Event of Default that
     has not been cured or waived to exercise the rights and powers vested in
     the Guarantee Trustee by this Guarantee, and to use the same degree of care
     and skill in exercising such rights and powers as a reasonably prudent
     person would use under the circumstances in the conduct of his own affairs;

         (vi) the Guarantee Trustee shall not be bound to make any investigation
     into the facts or matters stated in any resolution, certificate, statement,
     instrument, opinion, report, notice, request, direction, consent, order,
     bond, debenture, note, other evidence of indebtedness or other paper or
     document, but the Guarantee Trustee, in its discretion, may make such
     further inquiry or investigation into such facts or matters as it may see
     fit;

         (vii) the Guarantee Trustee may execute any of the trusts or powers
     hereunder or perform any duties hereunder either directly or by or through
     its agents or attorneys, and the Guarantee Trustee shall not be responsible
     for any negligence or willful misconduct on the part of any such agent or
     attorney appointed with due care by it hereunder. Nothing herein shall be
     construed as limiting or restricting the right of the Guarantor to bring
     any action directly against any agent or attorney appointed by the
     Guarantee Trustee for any negligence or willful misconduct on the part of
     such agent or attorney; and

                                       9
<PAGE>

         (viii) whenever in the administration of this Guarantee Agreement the
     Guarantee Trustee shall deem it desirable to receive instructions with
     respect to enforcing any remedy or right or taking any other action
     hereunder, the Guarantee Trustee (A) may request instructions from the
     Holders, (B) may refrain from enforcing such remedy or right or taking such
     other action until such instructions are received and (C) shall be fully
     protected in acting in accordance with such instructions.

     (b) No provision of this Guarantee Agreement shall be deemed to impose any
duty or obligation on the Guarantee Trustee to perform any act or acts or
exercise any right, power, duty or obligation conferred or imposed on it in any
jurisdiction in which it shall be illegal, or in which the Guarantee Trustee
shall be unqualified or incompetent in accordance with applicable law, to
perform any such act or acts or to exercise any such right, power, duty or
obligation. No permissive power or authority available to the Guarantee Trustee
shall be construed to be a duty to act in accordance with such power and
authority.

Section 3.3.  Indemnity.

  The Guarantor agrees to indemnify the Guarantee Trustee (which for purposes of
this Section 3.3 shall include its directors, officers, and employees) for, and
to hold the Guarantee Trustee harmless against, any loss, liability or expense
incurred without negligence, willful misconduct or bad faith on the part of the
Guarantee Trustee, arising out of or in connection with the acceptance or
administration of this Guarantee Agreement, including the reasonable costs and
expenses of defending against any claim or liability in connection with the
exercise or performance of any of its powers or duties hereunder.  The Guarantee
Trustee will not claim or exact any lien or charge on any Guarantee Payments as
a result of any amount due to it under this Guarantee Agreement.

Section 3.4.  Expenses.

  The Guarantor shall from time to time reimburse the Guarantee Trustee for its
reasonable expenses and costs (including reasonable attorneys' or agents' fees)
incurred in connection with the performance of its duties hereunder.

                                   ARTICLE IV

                               GUARANTEE TRUSTEE

Section 4.1.  Guarantee Trustee; Eligibility.

     (a) There shall at all times be a Guarantee Trustee which shall:

         (i) not be an Affiliate of the Guarantor; and

         (ii) be a Person that is eligible pursuant to the Trust Indenture Act
     to act as such and has a combined capital and surplus of at least
     $50,000,000, and shall be a corporation

                                       10
<PAGE>

     meeting the requirements of Section 310(a) of the Trust Indenture Act. If
     such corporation publishes reports of condition at least annually, pursuant
     to law or to the requirements of the supervising or examining authority,
     then, for the purposes of this Section and to the extent permitted by the
     Trust Indenture Act, the combined capital and surplus of such corporation
     shall be deemed to be its combined capital and surplus as set forth in its
     most recent report of condition so published.

     (b) If at any time the Guarantee Trustee shall cease to be eligible to so
act under Section 4.1(a), the Guarantee Trustee shall immediately resign in the
manner and with the effect set out in Section 4.2(b).

     (c) If the Guarantee Trustee has or shall acquire any "conflicting
interest" within the meaning of Section 310(b) of the Trust Indenture Act, the
Guarantee Trustee and Guarantor shall in all respects comply with the provisions
of Section 310(b) of the Trust Indenture Act.

Section 4.2.  Appointment, Removal and Resignation of the Guarantee Trustee.

     (a) No resignation or removal of the Guarantee Trustee and no appointment
of a Successor Guarantee Trustee pursuant to this Article IV shall become
effective until the acceptance of appointment by the Successor Guarantee Trustee
by written instrument executed by the Successor Guarantee Trustee and delivered
to the Holders and the Guarantee Trustee.

     (b) Subject to Section 4.2(a), a Guarantee Trustee may resign at any time
by giving written notice thereof to the Holders. The Guarantee Trustee shall
appoint a successor by requesting from at least three Persons meeting the
eligibility requirements such Person's expenses and charges to serve as the
Guarantee Trustee, and selecting the Person who agrees to the lowest expenses
and charges. If the instrument of acceptance by the Successor Guarantee Trustee
shall not have been delivered to the Guarantee Trustee within 60 days after the
giving of such notice of resignation, the Guarantee Trustee may petition, at the
expense of the Guarantor, any court of competent jurisdiction for the
appointment of a Successor Guarantee Trustee.

     (c) The Guarantee Trustee may be removed for cause at any time by Act
(within the meaning of Section 6.8 of the Trust Agreement) of the Holders of at
least a Majority in Liquidation Amount of the Preferred Securities, delivered to
the Guarantee Trustee.

     (d) If a resigning Guarantee Trustee shall fail to appoint a successor, or
if a Guarantee Trustee shall be removed or become incapable of acting as
Guarantee Trustee, or if any vacancy shall occur in the office of any Guarantee
Trustee for any cause, the Holders of the Preferred Securities, by Act of the
Holders of record of not less than 25% in aggregate Liquidation Amount of the
Preferred Securities then outstanding delivered to such Guarantee Trustee, shall
promptly appoint a successor Guarantee Trustee. If no Successor Guarantee
Trustee shall have been so appointed by the Holders of the Preferred Securities
and such appointment accepted by the Successor Guarantee Trustee, any Holder, on
behalf of himself and all others similarly situated, may petition any court of
competent jurisdiction for the appointment of a Successor Guarantee Trustee.

                                       11
<PAGE>

                                   ARTICLE V

                                   GUARANTEE

Section 5.1.  Guarantee.

  The Guarantor irrevocably and unconditionally agrees to pay in full on a
subordinated basis as set forth in Section 6.1 hereof to the Holders the
Guarantee Payments (without duplication of amounts theretofore paid by or on
behalf of the Issuer Trust), as and when due, regardless of any defense, right
of set-off or counterclaim which the Issuer Trust may have or assert, except the
defense of payment.  The Guarantor's obligation to make a Guarantee Payment may
be satisfied by direct payment of the required amounts by the Guarantor to the
Holders or by causing the Issuer Trust to pay such amounts to the Holders.  The
Guarantor shall give prompt written notice to the Guarantee Trustee in the event
it makes any direct payment hereunder.

Section 5.2.  Waiver of Notice and Demand.

  The Guarantor hereby waives notice of acceptance of the Guarantee Agreement
and of any liability to which it applies or may apply, presentment, demand for
payment, any right to require a proceeding first against the Guarantee Trustee,
the Issuer Trust or any other Person before proceeding against the Guarantor,
protest, notice of nonpayment, notice of dishonor, notice of redemption and all
other notices and demands.

Section 5.3.  Obligations Not Affected.

  The obligations, covenants, agreements and duties of the Guarantor under this
Guarantee Agreement shall in no way be affected or impaired by reason of the
happening from time to time of any of the following:

     (a) the release or waiver, by operation of law or otherwise, of the
performance or observance by the Issuer Trust of any express or implied
agreement, covenant, term or condition relating to the Preferred Securities to
be performed or observed by the Issuer Trust;

     (b) the extension of time for the payment by the Issuer Trust of all or any
portion of the Distributions (other than an extension of time for payment of
Distributions that results from the extension of any interest payment period on
the Junior Subordinated Debentures as so provided in the Indenture), Redemption
Price, Liquidation Distribution or any other sums payable under the terms of the
Preferred Securities or the extension of time for the performance of any other
obligation under, arising out of, or in connection with, the Preferred
Securities;

     (c) any failure, omission, delay or lack of diligence on the part of the
Holders to enforce, assert or exercise any right, privilege, power or remedy
conferred on the Holders pursuant to the terms of the Preferred Securities, or
any action on the part of the Issuer Trust granting indulgence or extension of
any kind;

                                       12
<PAGE>

     (d) the voluntary or involuntary liquidation, dissolution, sale of any
collateral, receivership, insolvency, bankruptcy, assignment for the benefit of
creditors, reorganization, arrangement, composition or readjustment of debt of,
or other similar proceedings affecting, the Issuer Trust or any of the assets of
the Issuer Trust;

     (e) any invalidity of, or defect or deficiency in, the Preferred
Securities;

     (f) the settlement or compromise of any obligation guaranteed hereby or
hereby incurred; or

     (g) any other circumstance whatsoever that might otherwise constitute a
legal or equitable discharge or defense of a guarantor (other than payment of
the underlying obligation), it being the intent of this Section 5.3 that the
obligations of the Guarantor hereunder shall be absolute and unconditional under
any and all circumstances.

  There shall be no obligation of the Holders to give notice to, or obtain the
consent of, the Guarantor with respect to the happening of any of the foregoing.

Section 5.4.  Rights of Holders.

  The Guarantor expressly acknowledges that: (a) this Guarantee Agreement will
be deposited with the Guarantee Trustee to be held for the benefit of the
Holders; (b) the Guarantee Trustee has the right to enforce this Guarantee
Agreement on behalf of the Holders; (c) the Holders of a Majority in Liquidation
Amount of the Preferred Securities have the right to direct the time, method and
place of conducting any proceeding for any remedy available to the Guarantee
Trustee in respect of this Guarantee Agreement or exercising any trust or power
conferred upon the Guarantee Trustee under this Guarantee Agreement; and (d) any
Holder may institute a legal proceeding directly against the Guarantor to
enforce its rights under this Guarantee Agreement, without first instituting a
legal proceeding against the Guarantee Trustee, the Issuer Trust or any other
Person.

Section 5.5.  Guarantee of Payment.

  This Guarantee Agreement creates a guarantee of payment and not of collection.
This Guarantee Agreement will not be discharged except by payment of the
Guarantee Payments in full (without duplication of amounts theretofore paid by
the Issuer Trust) or upon the distribution of Junior Subordinated Debentures to
Holders as provided in the Trust Agreement.

Section 5.6.  Subrogation.

  The Guarantor shall be subrogated to all rights (if any) of the Holders
against the Issuer Trust in respect of any amounts paid to the Holders by the
Guarantor under this Guarantee Agreement; provided, however, that the Guarantor
shall not (except to the extent required by mandatory provisions of law) be
entitled to enforce or exercise any rights which it may acquire by way of
subrogation or any indemnity, reimbursement or other agreement, in all cases as
a result of payment under this Guarantee Agreement, if at the time of any such
payment, any amounts are due

                                       13
<PAGE>

and unpaid under this Guarantee Agreement. If any amount shall be paid to the
Guarantor in violation of the preceding sentence, the Guarantor agrees to hold
such amount in trust for the Holders and to pay over such amount to the Holders.

Section 5.7.  Independent Obligations.

  The Guarantor acknowledges that its obligations hereunder are independent of
the obligations of the Issuer Trust with respect to the Preferred Securities and
that the Guarantor shall be liable as principal and as debtor hereunder to make
Guarantee Payments pursuant to the terms of this Guarantee Agreement
notwithstanding the occurrence of any event referred to in subsections (a)
through (g), inclusive, of Section 5.3 hereof.

                                   ARTICLE VI

                          COVENANTS AND SUBORDINATION

Section 6.1.  Subordination.

  This Guarantee Agreement will constitute an unsecured obligation of the
Guarantor and will rank subordinate and junior in right of payment to all Senior
Indebtedness of the Guarantor to the extent and in the manner set forth in the
Indenture with respect to the Junior Subordinated Debentures, and the provisions
of Article XIII of the Indenture will apply, mutatis mutandis, to the
obligations of the Guarantor hereunder.  The obligations of the Guarantor
hereunder do not constitute Senior Indebtedness of the Guarantor.

Section 6.2.  Pari Passu Guarantees.

  The obligations of the Guarantor under this Guarantee Agreement shall rank
pari passu with any similar guarantee agreements issued by the Guarantor on
behalf of the holders of preferred or capital securities issued by the Issuer
Trust and with any other security, guarantee or other obligation that is
expressly stated to rank pari passu with the obligations of the Guarantor under
this Guarantee Agreement, including without limiting the foregoing, the
guarantee agreement between Guarantor and _________, dated as of ___________,
relating to $________ aggregate liquidation amount of $________ preferred
securities issued by Provident Trust I.

                                  ARTICLE VII

                                  TERMINATION

Section 7.1.  Termination.

  This Guarantee Agreement shall terminate and be of no further force and effect
upon (a) full payment of the Redemption Price of all Preferred Securities, (b)
the distribution of Junior Subordinated Debentures to the Holders in exchange
for all of the Preferred Securities or (c) full payment of the amounts payable
in accordance with Article IX of the Trust Agreement upon liquidation of the
Issuer Trust.  Notwithstanding the foregoing, this Guarantee Agreement will

                                       14
<PAGE>

continue to be effective or will be reinstated, as the case may be, if at any
time any Holder is required to repay any sums paid with respect to the Preferred
Securities or this Guarantee Agreement.

                                  ARTICLE VIII

                                 MISCELLANEOUS

Section 8.1.  Successors and Assigns.

  All guarantees and agreements contained in this Guarantee Agreement shall bind
the successors, assigns, receivers, trustees and representatives of the
Guarantor and shall inure to the benefit of the Holders of the Preferred
Securities then outstanding.  Except in connection with a consolidation, merger
or sale involving the Guarantor that is permitted under Article VIII of the
Indenture and pursuant to which the assignee agrees in writing to perform the
Guarantor's obligations hereunder, the Guarantor shall not assign its
obligations hereunder, and any purported assignment that is not in accordance
with these provisions shall be void.

Section 8.2.  Amendments.

  Except with respect to any changes that do not materially adversely affect the
rights of the Holders (in which case no consent of the Holders will be
required), this Guarantee Agreement may only be amended with the prior approval
of the Holders of not less than a Majority in Liquidation Amount of the
Preferred Securities.  The provisions of Article VI of the Trust Agreement
concerning meetings of the Holders shall apply to the giving of such approval.

Section 8.3.  Notices.

  Any notice, request or other communication required or permitted to be given
hereunder shall be in writing, duly signed by the party giving such notice, and
delivered, telecopied (confirmed by delivery of the original) or mailed by first
class mail as follows:

     (a) if given to the Guarantor, to the address or telecopy number set forth
below or such other address or telecopy number or to the attention of such other
Person as the Guarantor may give notice to the Holders:

               Provident Bankshares Corporation
               114 East Lexington Street
               Baltimore, Maryland 21202
               Facsimile No.: (410) ___________
               Attention:  Office of the President

     (b) if given to the Issuer Trust, in care of the Guarantee Trustee, at the
Issuer Trust's (and the Guarantee Trustee's) address set forth below or such
other address or telecopy number or to the attention of such other Person as the
Guarantee Trustee on behalf of the Issuer Trust may give notice to the Holders:

                                       15
<PAGE>

               Provident Trust II
               Provident Bankshares Corporation
               114 East Lexington Street
               Baltimore, Maryland 21202
               Facsimile No.:  (410) _________
               Attention:  Office of the President


               with a copy to:
               Bankers Trust Company
               Four Albany Street - 4th Floor
               New York, New York  10006
               Facsimile No.:  (212) 250-6961
               Attention:  Corporate Trust and Agency Group;
               Corporate Market Services

     (c) if given to the Guarantee Trustee:

                Bankers Trust Company
                Four Albany Street - 4th Floor
                New York, New York  10006
                Facsimile No.: (212) 250-6961
                Attention:  Corporate Trust and Agency Group
                Corporate Market Services

     (d) if given to any Holder, at the address set forth on the books and
records of the Issuer Trust.

  All notices hereunder shall be deemed to have been given when received in
person, telecopied with receipt confirmed, or mailed by first class mail,
postage prepaid, except that if a notice or other document is refused delivery
or cannot be delivered because of a changed address of which no notice was
given, such notice or other document shall be deemed to have been delivered on
the date of such refusal or inability to deliver.

Section 8.4.  Benefit.

  This Guarantee Agreement is solely for the benefit of the Holders and is not
separately transferable from the Preferred Securities.

Section 8.5.  Interpretation.

  In this Guarantee Agreement, unless the context otherwise requires:

     (a) capitalized terms used in this Guarantee Agreement but not defined in
the preamble hereto have the respective meanings assigned to them in Section
1.1;

                                       16
<PAGE>

     (b) a term defined anywhere in this Guarantee Agreement has the same
meaning throughout;

     (c) all references to "the Guarantee Agreement" or "this Guarantee
Agreement" are to this Guarantee Agreement as modified, supplemented or amended
from time to time;

     (d) all references in this Guarantee Agreement to Articles and Sections are
to Articles and Sections of this Guarantee Agreement unless otherwise specified;

     (e) a term defined in the Trust Indenture Act has the same meaning when
used in this Guarantee Agreement unless otherwise defined in this Guarantee
Agreement or unless the context otherwise requires;

     (f) a reference to the singular includes the plural and vice versa; and

     (g) the masculine, feminine or neuter genders used herein shall include the
masculine, feminine and neuter genders.

Section 8.6.  Governing Law.

  THIS GUARANTEE AGREEMENT SHALL BE GOVERNED BY AND CONSTRUED AND INTERPRETED IN
ACCORDANCE WITH THE LAWS OF THE STATE OF NEW YORK WITHOUT REGARD TO THE CONFLICT
OF LAW PRINCIPLES THEREOF.

Section 8.7.  Counterparts.

  This instrument may be executed in any number of counterparts, each of which
so executed shall be deemed to be an original, but all such counterparts shall
together constitute but one and the same instrument.



                   [SIGNATURES APPEAR ON THE FOLLOWING PAGE]

                                       17
<PAGE>

  IN WITNESS WHEREOF, the parties hereto have caused this Guarantee Agreement to
be duly executed, all as of the day and year first above written.


                                          PROVIDENT BANKSHARES CORPORATION
                                          as Guarantor


                                          By:  _______________________________




                                          BANKERS TRUST COMPANY,
                                          as Guarantee Trustee
                                            and not in its individual capacity


                                          By:  _______________________________


                                       18