<PAGE>   1
EXHIBIT 10.3

                       COLLATERAL ASSIGNMENT OF FLOOD BOOK

         THIS COLLATERAL ASSIGNMENT is executed as of this 14 day August, 2001,
by Bankers Underwriters, Inc., a Florida corporation with its principal place of
business located in St. Petersburg, Florida (herein, "Pledgor").


                                R E C I T A L S:

         1. Insurance Management Solutions Group, Inc., a Florida corporation
with its principal place of business located in St. Petersburg, Florida,
(herein, "Lender") has established a line of credit (herein, "Credit Line") in
favor of Bankers Insurance Group, Inc., a Florida corporation (herein,
"Borrower"), in the amount of Five million ($5,000,000.00) dollars evidenced by
Borrower's Master Promissory Note (herein, "Note") of even date herewith and in
like principal amount.

         2. In connection with the establishment of the Credit Line, Borrower
and Pledgor have executed and delivered the following documents, all dated of
even date herewith (herein together with the within document shall be called,
"Loan Documents") - Note, Credit and Security Agreement, Absolute Assignment of
Flood Book (herein, "Absolute Assignment"), Further Assurances and Compliance
Agreement and Stock Option Agreement.

         3. Borrower, as the immediate parent of Pledgor, has agreed to advance
up to the entire amount of the proceeds of the Credit Line to Pledgor.

         4. Pledgor, as a Florida general insurance agent for Bankers Insurance
Company and First Community Insurance Company is the owner of the Flood Book.
For purposes hereof the Flood Book shall mean all of Pledgor's right, title and
interest in and to all of the following, whether now owned or hereafter
acquired, together with all replacements therefor and proceeds (including but
without limitation, insurance policies) thereof:

                  Accounts and contract rights, as those terms are defined by
                  the Uniform Commercial Code as adopted by the State of
                  Florida, with insurance agents, including but not limited to
                  general agents with respect to the sale of federal flood
                  insurance

         5. Lender has, as a condition precedent to establishing such line of
credit, required the execution of this Assignment.

         NOW, THEREFORE, in consideration of the aforesaid Credit Line as well
as for other good and valuable consideration, Pledgor hereby grants to Lender a
first lien security interest in and to the Flood Book to Lender as security for
the repayment of all sums advanced on the Credit Line and for the performance by
Borrower and Pledgor of all covenants and agreements under the Loan Documents in
accordance with the following terms and conditions. Unless the context shall
otherwise require, capitalized terms used and not defined herein shall have the
meanings assigned thereto in the Credit and Security Agreement dated of even
date herewith between Lender and Borrower (herein, "Credit Agreement").


<PAGE>   2



         1. POWER OF ATTORNEY. Pledgor irrevocably nominates, constitutes and
appoints Lender its true and lawful attorney in fact, with full power of
substitution and revocation for it, in its name, place and stead and either in
the name of Lender or in the name of Pledgor to assign any agency agreements
between Pledgor and agents responsible for the production of federal flood
insurance policies as well as any other instruments, agreements or certificates
necessary or appropriate to give full force an effect to such assignment which
assignment may be to Lender, to First Community Insurance Company or to such
other party as Lender may, in the exercise of its absolute discretion select
whether pursuant to the delivery of the Absolute Assignment or otherwise, but
Lender shall not be under any duty to exercise any such authority or power or in
any way be responsible for the collection of any sums owing under the Flood
Book. All rights, powers and authority of said attorney-in-fact to exercise any
and all rights and powers herein granted shall commence and be in full force and
effect as of the date of this Agreement and such rights, powers and authority
shall remain in full force and effect thereafter until the obligations of
Borrower and Pledgor under the Loan Documents have been satisfied in full
However, Lender shall not deliver the Absolute Assignment or otherwise exercise
its rights under this Power of Attorney until there is an occurrence of an Event
of Default hereunder. The power set forth herein is a power coupled with an
interest.

         2. AFFIRMATIVE COVENANTS. Pledgor hereby covenants with Lender that
until the obligations of Borrower and Pledgor under the Loan Documents have been
satisfied in full and for so long as any amount remains outstanding under the
Credit Line, Pledgor will keep the Flood Book free from all liens, encumbrances
and security interests and pay and discharge when due all taxes, levies and
other charges upon them and defend them against all claims of any kind.

         3. REPRESENTATIONS AND WARRANTIES. Pledgor represents and warrants to
Lender that it is the lawful owner and holder of the Flood Book described above,
that:

                (a)     Pledgor has good right to sell, transfer and assign the
                        same as aforesaid.

                (b)     There are no actions, suits or proceedings pending or,
                        to the knowledge of Pledgor, threatened against or
                        affecting it whether civil, criminal, administrative or
                        investigative, and it is not in default with respect to
                        any judgment, decision, order, writ, injunction, decree
                        or demand of any court or governmental authority;

                (c)     The consummation of the transactions hereby contemplated
                        in performance of this Agreement or of any of the Loan
                        Documents will not result in any breach of or constitute
                        a default under any mortgage, deed of trust, lien, bank
                        loan or credit agreement, corporate charter, by-law or
                        other instrument to which Pledgor is a party, or by
                        which it is bound or affected;

                (d)     Pledgor is a corporation duly organized, validly
                        existing and in good standing under the laws of the
                        State of Florida; Pledgor has all corporate powers to
                        own its properties and to engage in the business it
                        conducts, and is duly qualified and in good standing as
                        a foreign corporation in the jurisdictions wherein the
                        nature of the business transacted by it or property
                        owned by it make such qualification necessary;



<PAGE>   3

                (e)     All of the representations and warranties set forth in
                        this paragraph 3 shall survive the execution and
                        delivery of this Agreement and the other Loan Documents
                        until obligations of Borrower and Pledgor under the Loan
                        Documents have been satisfied in full and all
                        outstanding amounts due under the Credit Line are paid
                        in full.

         4. EVENTS OF DEFAULT. The phrase Event of Default shall have the
definition assigned to such phrase under the Credit Agreement, and any Event of
Default under the Credit Agreement shall constitute an Event of Default
hereunder.

         5. DEFAULT REMEDIES. Upon any Event of Default, all or any portion of
amounts due or to become due from Borrower to Lender under the Note, shall, at
the option of Lender, without demand, presentment or dishonor all of which
Pledgor hereby waives, become at once due and payable, and Lender shall
thereupon have all the rights and remedies of a secured party under the Uniform
Commercial Code as adopted by the State of Florida. The failure or delay of
Lender to exercise or enforce any rights, liens, powers or remedies hereunder or
under any of the aforesaid agreements or other documents of security or
Collateral shall not operate as a waiver of such liens, rights, powers and
remedies, but all such liens, rights, powers and remedies shall continue in full
force and effect until all loans and advances under the Credit Line shall have
been fully satisfied.

         6. TERMINATION OF AGREEMENT. This Agreement shall terminate upon the
satisfaction in full of all obligations of Borrower and Pledgor under the Loan
Documents, including but not limited to payment in full of all amounts
outstanding under the Loan Documents.

         7. MODIFICATION. No change or modification of this Agreement shall be
valid unless the same shall be in writing and signed by all of the parties
hereto.

         8. CAPTIONS. The paragraph captions as to contents of the particular
paragraphs herein are inserted only for convenience and are in no way to be
construed as part of this Agreement or as a limitation of the scope of the
particular paragraph in which they are referred.

         9. COUNTERPARTS. This Agreement may be executed in several
counterparts, each of which so executed shall deem to be an original and said
counterpart shall, together constitute and be one and the same instrument.

         10. CONSTRUCTION OF AGREEMENT. Words of a gender used in this Agreement
shall be held to include any other gender, the words in a singular number held
to include the plural, when the sentence so requires.

         11. INVALIDATION. Should any part of this Agreement for any reason be
declared invalid, such decision shall not effect the validity of any remaining
portion, which remaining portion shall remain in force and effect as if the
Agreement had been executed with the invalid portion thereof eliminated. It is,
therefore, declared the intention of the parties hereto that each of them will
have executed the remaining portion of this Agreement without including therein
any such part, parts or portion which may, for any reason, be hereafter declared
void.

         12. ATTORNEY'S FEES. Subject to reasonable construction and sound
business practices, if Borrower, Pledgor or Lender should bring a Court action
alleging breach of this Agreement or seeking to enforce, rescind, renounce,
declare, void or terminate this Agreement or



<PAGE>   4

any provisions thereof, the prevailing party shall be entitled to recover all of
its legal expenses, including reasonable attorney's fees and costs (including
legal expenses for any appeals taken), and to have the same awarded as part of
the judgment in the proceeding in which such legal expenses and attorney's fees
were incurred.

         13. APPLICABLE LAW/VENUE. This Agreement shall be construed in
accordance with and governed by the laws of the State of Florida, without regard
to choice of law provisions. Further, the venue for any action brought to
enforce any of the provisions hereof shall be in a state court of competent
jurisdiction in Pinellas County, Florida and any action commenced in any other
forum may be removed to a state court of competent jurisdiction in Pinellas
County, Florida.

         IN WITNESS WHEREOF, Borrower has set its hand and seal the day and year
first above written.

Signed sealed and delivered                     Bankers Insurance Group, Inc.
In the presence of:                             a Florida corporation


s/s Lisa M. Powell                              By:      G. Kristin Delano
------------------------------------                --------------------------
Signature                                       As its:  Secretary

s/s Harold David Holland
------------------------------------
Name legibly printed, typewritten or stamped

Signed sealed and delivered                     Bankers Underwriters, Inc.
In the presence of:                             a Florida corporation


s/s Lisa M. Powell                              By:      Richard Torra
------------------------------------                --------------------------
Signature                                       As its:  Corporate Secretary

s/s Harold David Holland
--------------------------------------------
Name legibly printed, typewritten or stamped