<PAGE>
 
                                                                   Exhibit 10.36

                              FOURTH AMENDMENT TO
                         SHAREHOLDER RIGHTS AGREEMENT
                         ----------------------------


     Fourth Amendment, dated as of February 23, 1999 (this "Fourth Amendment"),
to the Shareholder Rights Agreement, dated as of July 25, 1991, between Little
Switzerland, Inc., a Delaware corporation (the "Company"), and State Street Bank
and Trust Company, a Massachusetts trust company (the "Rights Agent"), as
amended by the First Amendment to Shareholder Rights Agreement, dated as of
November 8, 1991, between the Company and the Rights Agent, the Second Amendment
to Shareholder Rights Agreement, dated as of April 6, 1993, between the Company
and the Rights Agent and the Third Amendment to Shareholder Rights Agreement,
dated as of February 4, 1998, between the Company and the Rights Agent (the
"Rights Agreement").

                              W I T N E S S E T H
                              - - - - - - - - - -

     WHEREAS, in accordance with the terms of the Rights Agreement, the Company
deems it desirable to make certain amendments to the Rights Agreement; and

     WHEREAS, Section 27 of the Rights Agreement provides that prior to the
Distribution Date (as defined therein), the Company and the Rights Agent shall,
if the Company so directs, amend or supplement any provision of the Rights
Agreement as the Company may deem necessary or desirable without the approval of
holders of the Company's common stock, par value $.01 per share (the "Common
Stock");

     WHEREAS, the Company intends to enter into a Settlement Agreement (the
"Settlement Agreement") with Jewelcor Management, Inc., Seymour Holtzman,
ValueVest Partners, L.P., Donald L. Sturm, C. William Carey (collectively, the
"Stockholders") and certain other parties identified therein, pursuant to which,
among other things, the Stockholders will agree to vote or cause to be voted all
shares of the Common Stock beneficially owned by such Stockholders or over which
such Stockholders exercise voting control for the election of the nominees
designated by the Board of Directors at the Company's 1998 Annual Meeting of
Stockholders as set forth in such Settlement Agreement;

     WHEREAS, prior to entering into the Settlement Agreement, the Company
desires to amend certain provisions of the Rights Agreement;

     NOW, THEREFORE, in consideration of the premises and the mutual agreements
herein set forth, the parties hereby agree that the Rights Agreement is hereby
amended as follows:

     1.   Section 1(a) of the Rights Agreement is amended by adding at the end
of Section 1(a) a new paragraph which provides as follows:
<PAGE>
 
     Notwithstanding anything in this Agreement to the contrary, neither
Jewelcor Management, Inc. ("JMI"), Seymour Holtzman, ValueVest Partners, L.P.
("ValueVest"), Donald L. Sturm, C. William Carey, nor any of their Affiliates or
Associates, shall be deemed to be an Acquiring Person or an Adverse Person, and
no Stock Acquisition Date or Distribution Date shall occur, as a result of 
(i) the negotiation, execution and delivery of the Settlement Agreement, dated
as of February 23, 1999, by and among JMI, Seymour Holtzman, ValueVest, 
C. William Carey, the Company and the other parties identified therein (the
"Settlement Agreement"); or (ii) any action taken by JMI, Seymour Holtzman,
ValueVest, Donald L. Sturm, C. William Carey or any of their Affiliates or
Associates in accordance with the provisions of the Settlement Agreement.


                 [REMAINDER OF PAGE INTENTIONALLY LEFT BLANK.]
<PAGE>
 
     IN WITNESS WHEREOF, the parties hereto have caused this Fourth Amendment to
be duly executed as of the day and year first above written.  This Fourth
Amendment may be executed in one or more counterparts all of which shall be
considered one and the same amendment and each of which shall be deemed to be an
original.


                                    LITTLE SWITZERLAND, INC.

                                    By: /s/ Adriane J. Dudley
                                        -----------------------
                                        Name: Adriane J. Dudley
                                        Title: Director


                                    STATE STREET BANK AND TRUST
                                    COMPANY, as Rights Agent

                                    By: /s/ Charles Rossi
                                        -----------------------
                                      Name: Charles Rossi
                                      Title: President