<PAGE>   1

                                                                   EXHIBIT 10.54


                               SECOND AMENDMENT TO
                              EMPLOYMENT AGREEMENT


         THIS SECOND AMENDMENT is entered into as of April 1, 1998 by and
between IMC MORTGAGE COMPANY, a Florida corporation (the "Company") and STUART
D. MARVIN (the "Executive").

                                   Background

         A. Industry Mortgage Company, L.P., a Delaware limited partnership (the
"Limited Partnership") entered into an employment agreement dated as of August
1, 1996 with the Executive which is amended by a First Amendment thereto dated
October 3rd , 1997 pursuant to which all obligations of the limited partnership
were assumed by the Company and pursuant to which certain further changes were
made (which agreement as amended is called the "Existing Agreement"); and

         B. The parties wish to further amend the Existing Agreement to
incorporate the changes set forth below relating to the payment of deferred
compensation to the Executive.

         NOW, THEREFORE, in consideration of the mutual benefits to be derived
herefrom, the parties hereto agree that the Existing Agreement is hereby further
modified as follows:

         1. Automatic Renewal. The following provision is hereby added to
Section 3 of the Existing Agreement:

                     3.6 Automatic Extension. This Agreement shall be
         automatically extended for successive three (3) year periods at the end
         of the initial and each extended term thereafter, unless either party
         provides written notice of termination to the other party at least six
         (6) months prior to the expiration of the initial or such extended
         term, respectively. In the event the Company terminates this Agreement
         or fails to renew this Agreement or does not permit the automatic
         extension to occur at the end of any term hereof, Executive shall be
         entitled to receive his Deferred Compensation under section 4.2 hereof.


         2. Deferred Compensation. The following provision is hereby adopted in
substitution of Section 4.2 of the Existing Agreement ("Deferred Compensation")
and shall hereafter govern the provisions of the Agreement:

                     4.2  Deferred Compensation.

                          (a) When Due. Executive (or his estate as the case
                  may be) shall be 


                                       1


<PAGE>   2



                  entitled to the Deferred Compensation as calculated in the
                  event that Executive's employment is terminated for any of the
                  following reasons herein: (i) death of Executive; (ii)
                  termination by the Company without cause pursuant to Section
                  3.3; (iii) termination by Executive upon default by the
                  Company pursuant to Section 3.4; (iv) termination by Executive
                  after a Change of Control pursuant to Section 3.5; (vi)
                  termination by the Company pursuant to Section 3.6, or (vii)
                  termination by the Company pursuant to Section 5.1.

                           (b) Amount. The Deferred Compensation shall be the
                  amount ("Base Deferred Compensation") which is calculated as
                  the greater of (i) the Base Salary payments Executive would
                  have received had his employment continued for the remaining
                  term of this Agreement (including yearly increases calculated
                  at the maximum increase for the prior three years); or (ii) an
                  amount equal to 150% of the highest compensation earned by
                  Executive in any of the past three calendar years (including
                  both Base Salary and Bonus Compensation, including any
                  compensation paid under the Executive Officer Unregistered
                  Stock Plan), with compensation for calendar year 1997 being
                  treated as including Bonus Compensation equal to three hundred
                  percent of Executive's Base Salary is the Base Salary was at
                  the end of 1997. In addition to the Base Deferred
                  Compensation, Executive shall be entitled to the following
                  (which, together with the Base Deferred Compensation shall be
                  collectively called the "Deferred Compensation") all of the
                  benefits and personal perquisites otherwise provided in this
                  Agreement (including automobile expenses) during that period
                  of time which is the greater of (i) the remaining term of this
                  Agreement, or (ii) two years (the "Deferral Period"). The
                  Deferred Compensation herein shall be deemed liquidated
                  damages resulting from the Company's termination of this
                  Agreement and shall be Executive's sole and exclusive remedy
                  for any such termination. Deferred Compensation shall not be
                  diminished or offset by reason of any earnings by Executive
                  subsequent to the date of termination.

                           (c) Payment of Deferred Compensation. Except as
                  provided below, the Deferred Compensation shall be paid in
                  monthly installments over the thirty-six (36) months following
                  the event giving rise to a Deferred Compensation. If such
                  termination is a result of the death of Executive, the initial
                  Deferred Compensation shall be made within fifteen (15) days
                  after the personal representative of Executive's estate
                  notifies the Company that Letters of Administration have been
                  filed in the probate proceeding. The Company shall have the
                  option at all times during the term of this Agreement to
                  maintain key man life insurance on Executive's life to cover
                  the cost of any Deferred Compensation due to Executive. If
                  such key man life insurance is maintained, and the Deferred
                  Compensation is due as a result of Executive's death, the
                  Deferred Compensation shall be paid 100% in cash upon
                  Executive's death.

         3. Ratification as to Other Respects. The Existing Agreement is hereby
ratified and confirmed and remains in full force and effect in all respects
except as hereby modified.


                                       2


<PAGE>   3


         IN WITNESS WHEREOF, this Amendment has been executed as of the day and
year first above written.


                                       IMC MORTGAGE COMPANY


                                       By: /s/ George Nicholas
                                          ------------------------------------
                                          George Nicholas, Chairman


                                        /s/ Stuart D. Marvin
                                       ---------------------------------------
                                       STUART D. MARVIN, Executive






                                       3