<PAGE>   1
                                                                    EXHIBIT 10.4

                        THIRD AMENDMENT TO NOTE AGREEMENT

         THIS THIRD AMENDMENT, dated as of April 13, 2001 (this "Third
Amendment") to the Note Agreement dated as of January 15, 1996 (as amended by
the First Amendment to Note Agreement dated as of April 16, 1997 (the "First
Amendment"), and as further amended by the Second Amendment to Note Agreement
dated as of September 30, 1997 (the "Second Amendment") is made between ALLIED
HOLDINGS, INC., a Georgia corporation (the "Company"), and each of the
institutions which is a signatory to this Third Amendment (collectively, the
"Noteholders") and is effective as of December 31, 2000.

                                    RECITALS:

         A.       The Company and each of the Noteholders have heretofore
entered into the Note Agreement dated as of January 15, 1996, as amended by the
First Amendment and as further amended by the Second Amendment (as so amended,
the "Note Agreement"). The Company heretofore issued the $40,000,000 12% Senior
Subordinated Notes due February 1, 2003 (the "Notes") pursuant to the Note
Agreement.

         B.       The Company and the Noteholders now desire to amend the Note
Agreement in the respects, but only in the respects, hereinafter set forth.

         C.       Capitalized terms used herein shall have the respective
meanings ascribed thereto in the Note Agreement unless herein defined or the
context shall otherwise require.

         D.       All requirements of law have been fully complied with and all
other acts and things necessary to make this Third Amendment a valid, legal and
binding instrument according to the terms for the purposes herein expressed have
been done or performed.

         NOW, THEREFORE, upon the full and complete satisfaction of the
conditions precedent to the effectiveness of this Third Amendment set forth in
ss.3.1 hereof, and in consideration of other good and valuable consideration,
the receipt and sufficiency of which is hereby acknowledged, the Company and the
Noteholders do hereby agree as follows:

SECTION 1.        AMENDMENTS.

         1.1      Compliance with Section 5.6 of the Note Agreement shall be and
hereby is waived for the period commencing the date hereof through March 30,
2002. Commencing March 31, 2002, Section 5.6 of the Note Agreement shall be
deleted in its entirety and replaced with the following text:

<PAGE>   2






                  "Section 5.6. Consolidated Net Worth. The Company will at all
         times during the period commencing March 31, 2002 and ending June 29,
         2002 keep and maintain Consolidated Net Worth at an amount not less
         than the greater of (i) negative Four Million Dollars (-$4,000,000) or
         (ii) the sum of Seventeen Million Dollars ($17,000,000) plus the
         aggregate Consolidated Net Income for the immediately preceding four
         fiscal quarter period ending March 31, 2002 (such greater amount, the
         "Minimum Amount"), and for each period set forth in the table below
         shall keep and maintain Consolidated Net Worth at an amount equal to
         the sum of the Minimum Amount plus the amount set forth opposite the
         applicable period:

<TABLE>
<CAPTION>
                         Period                                        Amount
                         ------                                        ------
          <S>                                    <C>
          June 30, 2002 to September 29, 2002    50% of Consolidated Net Income for the Company's
                                                 fiscal quarter ending June 30, 2002 (but without
                                                 deduction in the event of a loss)

          September 30, 2002 to December 30,     50% of Consolidated Net Income for the Company's
          2002                                   two fiscal quarters ending September 30, 2002
                                                 (but without deduction in the event of a loss)

          December 31, 2002 and thereafter       50% of Consolidated Net Income for the Company's
                                                 three fiscal quarters ending December 31, 2002
                                                 (but without deduction in the event of a loss)
</TABLE>


         1.2      Section 5.7(a) of the Note Agreement shall be and hereby is
amended by inserting "for the incurrence of Indebtedness under the Credit
Agreement in excess of $200,000,000" after "2 to 1" where it appears in the
eleventh line of clause (5) (as added by the First Amendment).

         1.3      Compliance with Section 5.8 of the Note Agreement shall be and
hereby is waived for the period commencing the date hereof through March 30,
2002.

         1.4      Section 5.11 of the Note Agreement shall be and hereby is
amended by adding a new paragraph to the end thereof, to read as follows:

                  "Notwithstanding the foregoing provisions of this Section
         5.11, the Company shall not make or declare any Restricted Payments
         (for purposes of this sentence, the term Restricted Payment shall not
         include Restricted Investments)."

         1.5      Section 9.1 of the Note Agreement shall be and hereby is
amended as follows:


                                       2
<PAGE>   3

                  (a)      the definition of "Consolidated Cash Flow" is hereby
         amended by adding the following sentence to the end thereof:

                           "Notwithstanding anything to the contrary in this
                  definition, for the purpose of calculating the Fixed Charge
                  Coverage Ratio, the PIK Amount (as defined in the Notes) shall
                  not be included in the foregoing calculations.";

                  (b)      the definition of "Default Rate" is hereby amended
         and restated as follows:

                           "Default Rate" shall mean a rate of interest per
                  annum equal to the higher at the time of (i) the applicable
                  Accrual Rate (as defined in the Notes) plus 2% and (ii) the
                  rate announced by Fleet National Bank (or a successor thereto)
                  as its "Base Rate";

                  (c)      the definition of "Fixed Charges" is hereby amended
         by adding the following sentence to the end thereof:

                           "Notwithstanding anything to the contrary in this
                  definition, for the purpose of calculating the Fixed Charge
                  Coverage Ratio, the PIK Amount (as defined in the Notes) shall
                  not be included in the foregoing calculations.";

                  (d)      the definition of "Restricted Investments" is hereby
         amended by adding the following new subparagraph (j) immediately
         following subparagraph (i) thereof (as added by the First Amendment):

                           "(j) Investments in any Person, not otherwise
                  permitted by the provisions of subparagraphs (a) through (i)
                  above, inclusive, provided that (x) the aggregate amount of
                  such Investments shall not exceed Two Hundred Fifty Thousand
                  Dollars ($250,000) in the aggregate at any time outstanding,
                  in addition to the Investments permitted above; (y) at the
                  time of making each such Investment and after giving effect
                  thereto no Default or Event of Default shall have occurred and
                  be declared; and (z) the Noteholders agree that Investments
                  made under this subparagraph (j) shall not be deemed to be
                  Indebtedness under the Note Agreement."; and

                  (e)      the definition of "Revolving Credit Commitment
         Amount" is hereby amended by deleting "$130,000,000" and inserting
         "$230,000,000" in lieu thereof.

         1.6      Sections 11.4, 12.8, 12.9 and 12.10 of the Credit Agreement,
as in effect as of the date hereof after giving effect to Amendment No. 1 to the
Credit Agreement, dated as of April 13, 2001, by and among Allied Holdings,
Inc., Allied Systems (Canada) Company and the financial institutions and agents
named therein, but without giving effect to any subsequent amendments,
modifications, restatements or supplements thereto that have not been consented
to by each of the


                                       3
<PAGE>   4
Noteholders, are incorporated by reference as if fully set forth herein; except
that such incorporation by reference shall be subject to the following
modifications:

                  (a)      with respect to Section 11.4 of the Credit Agreement,
         as incorporated herein, references to the Banks shall be deemed to be
         references to the Noteholders;

                  (b)      with respect to Section 12.8 of the Credit Agreement,
         the table relating to Minimum Consolidated EBITDA as incorporated
         herein shall be modified to read as follows:

<TABLE>
<CAPTION>
              Month Ending                  Minimum Consolidated EBITDA
              ------------                  ---------------------------
           <S>                              <C>

             March 31, 2001                       $(13,700,000)
             April 30, 2001                       $ (5,000,000)
              May 31, 2001                        $  1,200,000
             June 30, 2001                        $  6,500,000
             July 31, 2001                        $          0
            August 31, 2001                       $  4,000,000
           September 30, 2001                     $  6,400,000
            October 31, 2001                      $ 17,600,000
           November 30, 2001                      $ 17,100,000
           December 31, 2001                      $ 18,300,000;
</TABLE>


                  (c)      with respect to Section 12.9 of the Credit Agreement,
         the ratio of (a) Consolidated Net Tangible Assets to (b) Consolidated
         Senior Secured Debt as incorporated herein shall be modified by
         deleting the words "not less than 1.90:1.00" and inserting the words
         "not less than 1.70:1.00" in lieu thereof; and

                  (d)      with respect to Section 12.10 of the Credit
         Agreement, the limits on Capital Expenditures as incorporated herein
         shall be modified by deleting the words:

                            "(a) during the fiscal quarter ending March 31,
                  2001, $8,300,000, (b) during the period of two fiscal quarters
                  ending June 30, 2001, $18,000,000, (c) during the period of
                  three fiscal quarters ending September 30, 2001, $21,500,000,
                  and (d) during the period of four fiscal quarters ending
                  December 31, 2001, $24,500,000",

                           and inserting in lieu thereof the words:

                           "(a) during the fiscal quarter ending March 31, 2001,
                  $9,300,000, (b) during the period of two fiscal quarters
                  ending June 30, 2001, $20,000,000, (c) during the period of
                  three fiscal quarters ending September 30, 2001, $24,500,000,


                                       4
<PAGE>   5

                  and (d) during the period of four fiscal quarters ending
                  December 31, 2001, $28,500,000".

SECTION 2.        REPRESENTATIONS AND WARRANTIES OF THE COMPANY.

         2.1      To induce the Noteholders to execute and deliver this Third
Amendment, the Company represents and warrants to each Noteholder (which
representations shall survive the execution and delivery of this Third
Amendment) that:

                  (a)      This Third Amendment has been duly authorized,
         executed and delivered by the Company and this Third Amendment
         constitutes the legal, valid and binding obligation, contract and
         agreement of the Company, enforceable against it in accordance with its
         terms, except as enforcement may be limited by bankruptcy, insolvency,
         reorganization, moratorium or similar laws or equitable principles
         relating to or limiting creditor's rights generally;

                  (b)      The Note Agreement, as amended by this Third
         Amendment, constitutes the legal, valid and binding obligations,
         contracts and agreements of the Company enforceable against it in
         accordance with their respective terms, except as enforcement may be
         limited by bankruptcy, insolvency, reorganization, moratorium or
         similar laws or equitable principles relating to or limiting creditors'
         rights generally; and

                  (c)      The execution, delivery and performance by the
         Company of this Third Amendment (i) has been duly authorized by all
         requisite corporate action and if required, shareholder action, (ii)
         does not require the consent or approval of any governmental or
         regulatory body or agency, and (iii) will not violate (x) any provision
         of law, statute, rule or regulation or its certificate of incorporation
         or bylaws, (y) any order of any court or any rule, regulation or order
         of any other agency or government binding upon it, or (z) any provision
         of any material indenture, agreement or other instrument to which it is
         a party or by which its properties or assets are or may be bound or
         result in a breach or constitute (alone or with due notice or lapse of
         time or both) a default thereunder; and (d) as of the date hereof and
         after giving effect to this Third Amendment, no Default or Event of
         Default has occurred which is continuing.

SECTION 3.        CONDITIONS TO EFFECTIVENESS OF THIS THIRD AMENDMENT.

         3.1      This Third Amendment shall not be effective until, and shall
become effective when, each and every one of the following conditions shall have
been satisfied:

                  (a)      Executed counterparts of this Third Amendment, duly
         executed by the Company and the holders of one hundred percent (100%)
         of the outstanding principal of the Notes, shall have been delivered to
         the Noteholders;


                                       5
<PAGE>   6

                  (b)      The Company shall have executed an Allonge to each of
         the outstanding Notes (each, an "Allonge"), substantially in the form
         attached hereto as Exhibit A, and each of the Notes, as modified by its
         respective Allonge, shall be in full force and effect;

                  (c)      The Company shall have entered into Amendment No. 1
         to the Credit Agreement, and all conditions to effectiveness of such
         Amendment No. 1 shall have been satisfied (other than any condition to
         effectiveness thereof which is conditioned on the effectiveness of this
         Third Amendment);

                  (d)      The Company shall have paid a fee (the "Restructuring
         Fee") to the Noteholders, in cash in the aggregate amount of $200,000,
         to be paid pro rata according to their respective principal amounts as
         set forth on Schedule I to the Note Agreement; and

                  (e)      The representations and warranties of the Company set
         forth in Section 2 hereof shall be true and correct on and with respect
         to the date hereof.

         Upon satisfaction of the foregoing conditions, this Third Amendment
shall become effective as of December 31, 2000.

SECTION 4.        REFERENCE TO AND EFFECT ON THE NOTE AGREEMENT.

         4.1      This Third Amendment shall be construed in connection with and
is a part of the Note Agreement, shall be effective as of December 31, 2000, and
except as modified and expressly amended by this Third Amendment and the
Allonges, all terms, conditions and covenants contained in the Note Agreement
and the Notes are hereby ratified and confirmed and shall be and remain in full
force and effect.

         4.2      Except as expressly provided herein, the execution, delivery
and effectiveness of this Third Amendment shall not operate as a waiver of: (i)
any right, power or remedy of the Noteholders under the Note Agreement or any of
the documents or agreements executed and delivered in connection therewith, or
(ii) any Event of Default under the Note Agreement.

         4.3      Upon the effectiveness of this Third Amendment, each reference
in the Note Agreement to:

                  (a)      "this Agreement," "hereunder," "hereof," "herein" or
         words of like import shall mean and be a reference to the Note
         Agreement as amended hereby;

                  (b)      "Notes" shall be to the Notes as amended hereby and
         by the Allonges.

SECTION 5.        MISCELLANEOUS.

         5.1      Any and all notices, requests, certificates and other
instruments executed and delivered after the execution and delivery of this
Third Amendment may refer to the Note


                                       6
<PAGE>   7

Agreement without making specific reference to this Third Amendment but
nevertheless all such references shall include this Third Amendment unless the
context otherwise requires.

         5.2      The provisions of this Third Amendment shall inure to the
benefit of, and be binding upon, each of the parties hereto and their respective
successors and assigns, including, without limitation, the holders from time to
time of the Notes and any interest therein.

         5.3      The descriptive headings of the various Sections or parts of
this Third Amendment are for convenience only and shall not affect the meaning
or construction of any of the provisions hereof.

         5.4      This Third Amendment shall be governed by and construed in
accordance with Illinois law.

         5.5      This Third Amendment may be executed in any number of
counterparts, each executed counterpart constituting an original, but all of
which together shall constitute only one agreement. Delivery of an executed
counterpart to this Third Amendment by telecopier shall be effective as delivery
of a manually executed counterpart to this Third Amendment.


                                       7
<PAGE>   8



         IN WITNESS WHEREOF, the Company and the Noteholders have caused this
Third Amendment to be duly executed and delivered all as of the day and year
first above written.


                                          ALLIED HOLDINGS, INC.



                                          By
                                            ----------------------------
                                              Name:
                                              Title:


                                          JOHN HANCOCK LIFE INSURANCE
                                            COMPANY (formerly John Hancock
                                            Mutual Life Insurance Company)



                                          By
                                            ----------------------------
                                              Name:
                                              Title:


                                          BARNETT & CO.



                                          By
                                            ----------------------------
                                              Name:
                                              Title:


                                          THE NORTHWESTERN MUTUAL LIFE
                                            INSURANCE COMPANY



                                          By
                                            ----------------------------
                                              Name:
                                              Title:


                                       8
<PAGE>   9

                                    EXHIBIT A

                                     ALLONGE
                                       TO
                12% SENIOR SUBORDINATED NOTE DUE FEBRUARY 1, 2003
                                       OF
                              ALLIED HOLDINGS, INC.

                                                                  April 13, 2001

         The 12% Senior Subordinated Note due February 1, 2003 of Allied
Holdings, Inc., Note number _______, issued in the original principal amount of
$[___________], to which this Allonge is attached, is hereby modified by the
terms hereof.

         Such Note is one of a series of Senior Subordinated Notes (herein
called the "Notes", the holders of such Notes, the "Noteholders") issued
pursuant to the Note Agreement, dated as of January 15, 1996, between Allied
Holdings, Inc. (the "Company") and the purchasers named therein, as amended by
(i) First Amendment to Note Agreement, dated as of April 16, 1997, by and among
the Company and the Noteholders, (ii) Second Amendment to Note Agreement, dated
as of September 30, 1997, by and among the Company and the Noteholders and (iii)
Third Amendment to Note Agreement, dated as of April 13, 2001, by and among the
Company and the Noteholders (as from time to time amended, the "Note
Agreement").

         All references in the Note to the Note Agreement shall be deemed and
construed to be references to the Note Agreement as defined in this Allonge.
Capitalized terms used herein without definition shall have the respective
meanings ascribed thereto in the Note Agreement.

         Interest shall be payable on this Note as follows:

         From and after April 13, 2001, interest (computed on the basis of a
360-day year of twelve 30-day months) shall accrue on the outstanding principal
amount of the Note not then overdue at the Accrual Rate (as defined below).

         On any interest payment date, so long as no Default or Event of Default
has occurred and is continuing, the Company may, upon at least three Business
Days and no more than ten Business Days prior written notice to the holder of
the Note, at its option defer to the maturity date of the Note an amount of
interest (such amount being referred to as the "PIK Amount") equal to (i) the
amount of the interest due on such interest payment date on the Note at the
applicable Accrual Rate, less (ii) the amount of interest that would accrue on
the Note at the Initial Rate to such interest payment date, provided that if the
Company exercises its option to defer interest on the Note, it shall

<PAGE>   10

contemporaneously exercise its interest deferral option with respect to all the
Notes. Each such PIK Amount shall be added to the outstanding principal amount
of the Note and interest on such amount shall accrue and be payable as
aforesaid. The holder hereof may enter the amount of such PIK Amount opposite
the applicable interest payment date on the schedule attached hereto as Annex I,
but the failure to enter such amount of PIK interest shall not affect the
outstanding principal amount of the Note or the obligation of the Company to pay
such amount, together with interest thereon as modified by this Allonge, and if
applicable the Make-Whole Amount.

         As used herein, the following terms shall have the following
definitions:

         "Accrual Rate" means a rate per annum equal to the Initial Rate plus
the Applicable Margin.

         "Applicable Margin" means:



<TABLE>
<CAPTION>
                    PERIOD                                 AMOUNT
                    ------                                 ------
    <S>                                                    <C>
     April 13, 2001 through June 30, 2001                   2.00%
    July 1, 2001 through September 30, 2001                 2.25%
   October 1, 2001 through December 31, 2001                2.50%
    January 1, 2002 through March 31, 2002                  2.75%
         April 1, 2002 and thereafter                       3.00%
</TABLE>


         "Initial Rate" means 12% per annum.

         "Default Rate" means a rate of interest per annum equal to the higher
at the time of (i) the applicable Accrual Rate (as defined in the Notes) plus 2%
and (ii) the rate announced by Fleet National Bank (or a successor thereto) as
its "Base Rate".


                                       10
<PAGE>   11



         Except as expressly provided herein, this Note is not modified or
amended in any respect and remains in full force and effect.



                                             ALLIED HOLDINGS, INC.


                                             By
                                               -------------------------
                                                 Name:
                                                 Title:



                                       11
<PAGE>   12


                                     ANNEX I



<TABLE>
<CAPTION>
                INTEREST PAYMENT DATE                  PIK AMOUNT
                ---------------------                  ----------
                <S>                                    <C>
                    August 1, 2001
                   February 1, 2002
                    August 1, 2002
                   February 1, 2003
</TABLE>