<PAGE>   1


                                                                     EXHIBIT 4.1

                      FOURTH AMENDMENT TO RIGHTS AGREEMENT

         THIS FOURTH AMENDMENT TO RIGHTS AGREEMENT (this "Amendment") is made as
of the 13th day of April, 2000, by and between RAILAMERICA, INC., a Delaware
corporation (the "Company"), and AMERICAN STOCK TRANSFER & TRUST COMPANY (the
"Rights Agent").



                                    RECITALS

         WHEREAS, on January 6, 1998, the Board of Directors of the Company
authorized and declared a dividend distribution of one Right for each share of
common stock, $.001 par value, of the Company outstanding at the close of
business on January 20, 1998; and

         WHEREAS, the Company and the Rights Agent entered into a certain Rights
Agreement, dated as of January 6, 1998, as amended September 3, 1998, April 18,
1999 and January 13, 2000 (the "Rights Agreement"), providing, among other
things, for the issuance of the Rights (all capitalized terms used in this
Amendment and not otherwise defined herein shall have the meanings ascribed to
such terms in the Rights Agreement); and

         WHEREAS, the Board of Directors of the Company has approved the
amendment of the Rights Agreement as hereinafter set forth, pursuant to
resolutions duly adopted at a meeting of the Board of Directors held on April
13, 2000;

         NOW, THEREFORE, in consideration of the premises and for other good and
valuable consideration, the receipt and sufficiency of which are hereby
acknowledged, the parties hereto agree as follows:

         1. Section 1 of the Rights Agreement is hereby amended by deleting the
definition of "Acquiring Person" set forth in subsection (a) thereof in its
entirety and substituting therefor a new definition of "Acquiring Person" to
read as follows:

                  "Acquiring Person" shall mean any Person who or which,
         together with all Affiliates and Associates of such Person, shall be
         the Beneficial Owner of fifteen percent (15%) or more of the shares of
         Common Stock then outstanding. Notwithstanding the foregoing, the term
         "Acquiring Person" shall not include (i) the Company, any Subsidiary of
         the Company, any employee benefit plan of the Company or of any
         Subsidiary of the Company, or any Person or entity organized, appointed
         or established by the Company for or pursuant to the terms of any such
         plan, (ii) EGS Associates, L.P., a Delaware limited partnership, EGS
         Partners L.L.C., a Delaware limited liability company, Bev Partners,
         L.P., a Delaware limited partnership, Jonas Partners, L.P., a New York
         limited partnership, EGS






<PAGE>   2

         Management, L.L.C., a Delaware limited liability company, William
         Ehrman, Frederic Greenberg, Jonas Gerstl and Julia Oliver, unless (a)
         such Persons shall be the Beneficial Owners, individually or in the
         aggregate, of more than twenty-five percent (25%) of the shares of
         Common Stock then outstanding, (b) the Agreement, dated August 21,
         1998, between the Company, EGS Associates, L.P., EGS Partners, L.L.C.,
         BEV Partners, L.P., Jonas Partners, L.P., William Ehrman, Frederic
         Greenberg, Frederick Ketcher, Jonas Gerstl, James McLaren and William
         Lautman, as such Agreement may be amended or supplemented from time to
         time, shall be determined to be unenforceable by a court of competent
         jurisdiction or (c) such Persons shall challenge the enforceability of
         such Agreement, (iii) Barclays Bank PLC and the lenders named in that
         certain Senior Secured Loan Facility and Guaranty Agreement (the
         "Facility") dated as of April 30, 1999 among Freight Victoria, the
         Company, RailAmerica Australia Pty Ltd., Barclays Bank PLC and the
         lenders named therein (such lenders collectively with Barclays Bank PLC
         to be referred to hereinafter as the "Lenders"), but only with respect
         to (A) warrants (the "Warrants") to purchase shares of Common Stock
         issued by the Company to the Lenders pursuant to the Facility and
         pursuant to that certain Warrant Agreement dated as of April 30, 1999
         between the Company and the Lenders and (B) the shares of Common Stock
         issuable upon exercise of the Warrants, it being understood that all
         other shares of Common Stock acquired by the Lenders independent of (A)
         or (B) above shall be considered for purposes of determining Acquiring
         Person status of Lenders; or (iv) any Person who or which, together
         with all Affiliates and Associates of such Person, would be an
         Acquiring Person solely by reason of a reduction in the number of
         issued and outstanding shares of Common Stock of the Company pursuant
         to a transaction or series of related transactions approved by a
         majority of the Independent Directors, if any, then in office and
         approved by a Supermajority Vote; provided, further, however, that in
         the event that such Person described in the foregoing clause (iv) does
         not become an Acquiring Person by reason of the foregoing clause (iv),
         such Person shall nonetheless become an Acquiring Person in the event
         such Person thereafter acquires Beneficial Ownership of an additional
         one percent (1%) of the Common Stock of the Company, unless such
         additional Common Stock ownership results solely from a subsequent
         reduction in the number of issued and outstanding shares of Common
         Stock of the Company."

         2. Except as specifically amended hereby, the Rights Agreement is and
remains unmodified and in full force and effect and is hereby ratified and
confirmed.

         3. This Amendment shall be deemed a contract made under the laws of the
State of Delaware and for all purposes shall be governed by and construed in
accordance with the laws of such State applicable to contracts made and to be
performed entirely within such State.

         4. This Amendment may be executed in counterparts and both of such
counterparts shall for all purposes be deemed to be an original, and such
counterparts shall constitute but one and the same instrument.









<PAGE>   3

         IN WITNESS WHEREOF, the parties hereto have caused this Amendment to be
duly executed as of the date first above written.



                                     RAILAMERICA, INC.


                                     By: /s/ Gary O. Marino
                                        ----------------------------------



                                     AMERICAN STOCK TRANSFER & TRUST COMPANY


                                     By: /s/ Herbert J. Lemmer
                                        ----------------------------------
                                             Herbert J. Lemmer
                                             Vice-President