<PAGE>   1
                                                                   EXHIBIT 10.36

                    AMENDED AND RESTATED FIFTH AMENDMENT TO
          AMENDED AND RESTATED LOAN AND SECURITY AGREEMENT AND CONSENT


               THIS AMENDED AND RESTATED FIFTH AMENDMENT TO AMENDED AND RESTATED
LOAN AND SECURITY AGREEMENT AND CONSENT (this "Amendment"), dated as of January
10, 2001, is entered into between CONGRESS FINANCIAL CORPORATION (WESTERN), a
California corporation ("Lender"), on the one hand, and WHEREHOUSE
ENTERTAINMENT, INC., a Delaware corporation, WHEREHOUSE.COM, INC., a California
corporation, WHEREHOUSE SUBSIDIARY I CO., INC., a Delaware corporation,
WHEREHOUSE SUBSIDIARY II CO., INC., a California corporation, and WHEREHOUSE
SUBSIDIARY III CO., INC., a Delaware corporation (collectively, "Borrower"), on
the other.

                                    RECITALS

               A. Borrower and Lender have previously entered into an Amended
and Restated Loan and Security Agreement dated as of October 26, 1998, as
amended by a First Amendment dated as of November 30, 1998, by a Second
Amendment dated as of May 14, 1999, by a Third Amendment dated as of August 31,
1999, by a Fourth Amendment dated as of November 16, 1999 and by a Fifth
Amendment (the "Fifth Amendment") dated as of December 6, 2000 (as amended, the
"Loan Agreement"), pursuant to which Lender has made certain loans and financial
accommodations available to Borrower. Terms used herein without definition shall
have the meanings ascribed to them in the Loan Agreement.

               B. Borrower and Lender wish to amend and restate the Fifth
Amendment on the terms and conditions set forth in this Amendment. Borrower and
Lender previously entered into that certain Fifth Amendment to Amended and
Restated Loan and Security Agreement dated as of December 6, 2000 (the "Prior
Amendment"). This Amendment amends, restates and supersedes in its entirety the
Prior Amendment. Borrower is entering into this Amendment with the understanding
and agreement that, except as specifically provided herein, none of Lender's
rights or remedies as set forth in the Loan Agreement is being waived or
modified by the terms of this Amendment.

               NOW, THEREFORE, in consideration of the foregoing and the mutual
covenants herein contained, and for other good and valuable consideration, the
receipt and sufficiency of which are hereby acknowledged, the parties hereby
agree as follows:

        1. Maximum Credit. "Maximum Credit" shall mean, with reference to the
Revolving Loans and the Letter of Credit Accommodations, the amount of One
Hundred Sixty-Five Million Dollars ($165,000,000); provided, however, that on
October 27, 2001, such amount shall be permanently reduced to One Hundred
Fifty-Five Million Dollars ($155,000,000).

        2. Used Inventory; Eligible Used Inventory. The term "Used Inventory"
shall mean Inventory consisting of used merchandise held for retail sale in the
ordinary course of the business of Borrower. The term "Eligible Used Inventory"
shall mean Used Inventory,


<PAGE>   2

consisting of audio recordings on compact disk, that would otherwise qualify as
Eligible Inventory, bur for the fact that such Inventory is in used condition.
Any Used Inventory that is not Eligible Used Inventory shall nevertheless be
part of the Collateral.

        3. Definition of Eligible Inventory. Section 1.20, the definition of
"Eligible Inventory" is amended to read in its entirety as follows:

               "1.20 "Eligible Inventory" shall mean Inventory consisting of
               finished merchandise held for sale in the ordinary course of the
               business of Borrower that is located at one of Borrower's retail
               stores, its distribution centers in the United States, a third
               party return processing facility, or in transit between any such
               location of Borrower, provided that, in any case, such location
               is in a jurisdiction where Lender has a first priority security
               interest in the Collateral, and is acceptable to Lender based on
               the criteria set forth below. In general, Eligible Inventory
               shall not include (without duplication in the following
               exclusions for the effect of any Availability Reserves) (a) spare
               parts for equipment; (b) packaging and shipping materials; (c)
               supplies used or consumed in Borrower's business; (d) Inventory
               at premises that are not owned by Borrower, unless, with respect
               to leased premises (other than a retail store), Lender has
               received a landlord waiver and, with respect to locations where
               Inventory is held by third parties, Lender has received a bailee
               waiver, in each case acknowledging the priority of Lender's liens
               and allowing Lender access to the premises for purposes of
               dealing with the Collateral, provided that, Borrower shall not be
               required to provide a landlord waiver with respect to the
               distribution center subject to that certain Transition Services
               Agreement dated as of August 10, 1998, until September 30, 1999;
               (e) Inventory in transit (other than Inventory in transit from
               one location of Borrower to another location satisfying the
               requirements set forth in this Section 1.20); (f) Inventory
               subject to a security interest or lien in favor of any person
               other than Lender except those permitted in this Agreement; (g)
               Inventory that is not subject to the first priority, valid and
               perfected security interest of Lender; (h) damaged and/or
               defective Inventory; (i) Inventory consisting of demos; (j) Used
               Inventory, other than Eligible Used Inventory; (k) Rental
               Merchandise; (l) Inventory purchased or sold on consignment; and
               (m) Inventory that is not actively held by Borrower for sale to
               retail customers, except for WEI Eligible Return Inventory and
               Operating Subsidiary Eligible Return Inventory. General criteria
               for Eligible Inventory may be established and revised from time
               to time by Lender in its reasonable credit judgment. Any
               Inventory that is not Eligible Inventory shall nevertheless be
               part of the Collateral."

        4. Lending Formulas. Section 2.1(a) of the Loan Agreement is amended to
read in its entirety as follows:

               "2.1 Revolving Loans.

               (a) Subject to, and upon the terms and conditions contained
herein, Lender agrees to make Revolving Loans (the "Tranche A Line") to Borrower
from time to time in


                                       2
<PAGE>   3

amounts requested by Borrower up to the lesser of One Hundred Fifty-five Million
Dollars ($155,000,000) or an amount equal to:

                             (1) The lesser of: (A) sixty-two percent (62%)
(sixty-five percent (65%) during the Seasonal Period) of the Value of Eligible
Inventory, other than Eligible Used Inventory, or (B) ninety percent (90%) of
the Net Recovery Cost Percentage multiplied by the Value of the Eligible
Inventory, plus

                             (2) The lesser of (A) fifty percent (50%) of the
Value of Eligible Used Inventory up to a maximum of $25,000,000 (not to exceed
15% of total Inventory), or (B) eighty-five percent (85%) of the Net Recovery
Cost Percentage multiplied by the Value of the Eligible Used Inventory, up to a
maximum of $25,000,000 (not to exceed 15% of total Inventory), minus

                             (3) the then undrawn amounts of outstanding Letter
of Credit Accommodations, minus

                             (4) any Availability Reserves.

The Revolving Loans made under Section 2.1(a) shall together be referred to as
the "Tranche A Loans".

        5. Fees.

               (a) Additional Fees. Borrower shall pay the following fees in
connection with this Amendment, in addition to the fees specified in the Loan
Agreement:

                      (i) Amendment Closing Fee. Borrower shall pay an amendment
closing fee of $387,500, which fee shall be fully earned upon execution of this
Amendment and shall be payable one-half on October 26, 2001 and one-half on
October 31, 2002.

                      (ii) Syndication Fee. Borrower shall pay a syndication fee
of $112,500, which fee shall be fully earned upon execution of this Amendment
and shall be payable on October 26, 2001.

               (b) Early Termination Fee. In lieu of the early termination fee
provided in Section 12.1(c) of the Loan Agreement, Borrower shall pay an early
termination fee as follows if the Loan Agreement is terminated in the periods
specified below:

<TABLE>
<CAPTION>
                           Amount                          Period
                           ------                          ------
<S>                                           <C>
    (i)     0.50% of the Maximum Credit       from the date of this Amendment to
                                              and including October 26, 2001
    (ii)    $300,000                          After October 26, 2001 to and
                                              including October 31, 2002
</TABLE>



                                       3
<PAGE>   4

<TABLE>
<CAPTION>
            Amount                                      Period
            ------                                      ------
<S>                                           <C>
    (iii)   $150,000                          After October 31, 2002 to and
                                              including October 31, 2003
</TABLE>

        6. Amendment to Term of Loan Agreement: The Renewal Date as set forth in
Section 12.1 of the Loan Agreement shall be October 31, 2003; provided, however,
that such renewal shall not apply to the Tranche B Line, and Lender shall not be
obligated to make any Tranche B Loans on or after October 26, 2001.

        7. Effectiveness of this Amendment. Lender must have received the
following items, in form and content acceptable to Lender, before this Amendment
is effective and before Lender is required to extend any credit to Borrower as
provided for by this Amendment:

               (a) Amendment. This Amendment fully executed in a sufficient
number of counterparts for distribution to Lender and Borrower;

               (b) Authorizations. Evidence that the execution, delivery and
performance by Borrower and each guarantor or subordinating creditor of this
Amendment and any instrument or agreement required under this Amendment have
been duly authorized;

               (c) Representations and Warranties. The representations and
warranties set forth in the Loan Agreement must be true and correct;

               (d) Consents. Counterparts of the Consent appended hereto (the
"Consent") executed on behalf of each of Wherehouse Holding I Co., Inc., a
Delaware corporation and Wherehouse Holding II Co., Inc., a Delaware corporation
("Guarantors", and together with Borrower, each a "Loan Party" and collectively
the "Loan Parties");

        8. Representations and Warranties. Borrower represents and warrants as
follows:

               (a) Authority. Each Loan Party has the requisite corporate power
and authority to execute and deliver this Amendment or the Consent, as
applicable, and to perform its obligations hereunder and under the Financing
Agreements (as amended or modified hereby) to which it is a party. The
execution, delivery and performance by Borrower of this Amendment and by each
other Loan Party of the Consent, and the performance by each Loan Party of each
Financing Agreement (as amended or modified hereby) to which it is a party have
been duly approved by all necessary corporate action of such Loan Party and no
other corporate proceedings on the part of such Loan Party are necessary to
consummate such transactions.

               (b) Enforceability. This Amendment has been duly executed and
delivered by Borrower. The Consent has been duly executed and delivered by each
Guarantor. This Amendment and each Financing Agreement (as amended or modified
hereby) is in full force and effect (subject to the terms of Section 7 hereof)
and is the legal, valid and binding obligation of


                                       4
<PAGE>   5

each Loan Party hereto or thereto, enforceable against such Loan Party in
accordance with its terms, and is in full force and effect.

               (c) Representations and Warranties. The representations and
warranties contained in each Financing Agreement (other than any such
representations or warranties that, by their terms, are specifically made as of
a date, or otherwise relate to a date, other than the date hereof) are correct
on and as of the date hereof as though made on and as of the date hereof.

               (d) No Default. No event has occurred and is continuing that
constitutes an Event of Default.

               (e) No Tranche B Loans. No Tranche B Loans are presently
outstanding.

        9. Governing Law. The validity of this Amendment, its construction,
interpretation and enforcement, the rights of the parties hereunder, shall be
determined under, governed by, and construed in accordance with the internal
laws of the State of California governing contracts only to be performed in that
State.

        10. Counterparts. This Amendment may be executed in any number of
counterparts and by different parties and separate counterparts, each of which
when so executed and delivered, shall be deemed an original, and all of which,
when taken together, shall constitute one and the same instrument. Delivery of
an executed counterpart of a signature page to this Amendment or the Consent by
telefacsimile shall be effective as delivery of a manually executed counterpart
of this Amendment or the Consent.

        11. Reference to and Effect on the Financing Agreements.

               (a) Upon and after the effectiveness of this Amendment, each
reference in the Loan Agreement to "this Agreement", "hereunder", "hereof" or
words of like import referring to the Loan Agreement, and each reference in the
other Financing Agreements to "the Loan Agreement", "thereof" or words of like
import referring to the Loan Agreement, shall mean and be a reference to the
Loan Agreement as modified and amended hereby.

               (b) Except as specifically amended above, the Loan Agreement and
all other Financing Agreements, are and shall continue to be in full force and
effect and are hereby in all respects ratified and confirmed and shall
constitute the legal, valid, binding and enforceable obligations of Borrower to
Lender.

               (c) The execution, delivery and effectiveness of this Amendment
shall not, except as expressly provided herein, operate as a waiver of any
right, power or remedy of Lender under any of the Financing Agreements, nor
constitute a waiver of any provision of any of the Financing Agreements.




                                       5
<PAGE>   6

               IN WITNESS WHEREOF, the parties have entered into this Amendment
as of the date first above written.

                                      CONGRESS FINANCIAL CORPORATION (WESTERN)


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________

                                      WHEREHOUSE ENTERTAINMENT, INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________

                                      WHEREHOUSE.COM, INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________

                                      WHEREHOUSE SUBSIDIARY I CO., INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________

                                      WHEREHOUSE SUBSIDIARY II CO., INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________

                                      WHEREHOUSE SUBSIDIARY III CO., INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________



                                       6
<PAGE>   7

                                     CONSENT



                          Dated as of January 10, 2001



               The undersigned, as Guarantors under their respective Guarantee,
each dated as of October 26, 1998 (as such terms are defined in and under the
Loan Agreement referred to in the foregoing Amendment), each hereby consents and
agrees to said Amendment and hereby confirms and agrees that its respective
Guarantee is, and shall continue to be in, in full force and effect and is
hereby ratified and confirmed in all respects except that, upon the
effectiveness of, and on and after the date of said Amendment, each reference in
each such Guarantor's Guarantee to the "Loan Agreement", "thereunder", "thereof"
or words of like import referring to the Loan Agreement, shall mean and be a
reference to the Loan Agreement as amended or modified by the said Amendment.



                                      WHEREHOUSE HOLDING I CO., INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________



                                      WHEREHOUSE HOLDING II CO., INC.


                                      By:_____________________________________
                                      Name:___________________________________
                                      Title:__________________________________





                                       7