<PAGE>

                                                                    EXHIBIT 10.4






================================================================================



                            LITTLE SWITZERLAND, INC.

                           INVESTOR'S RIGHTS AGREEMENT

                                   May 1, 2001



================================================================================

<PAGE>

                           INVESTOR'S RIGHTS AGREEMENT


                  This INVESTOR'S RIGHTS AGREEMENT (this "Agreement") is made as
of May 1, 2001 between Little Switzerland, a Delaware corporation (the
"Company"), and Jewelcor Management, Inc., a Nevada corporation (the
"Stockholder").

                                    RECITALS:

                  WHEREAS, concurrently with the execution of this Agreement,
the Company is agreeing to sell forty-five percent of its Common Stock, par
value $.01 per share, to Tiffany & Co. International, Inc., a Delaware
corporation ("Tiffany"), pursuant to the certain Stock Purchase Agreement, dated
of even date herewith (the "Stock Purchase Agreement"), between the Company and
Tiffany;

                  WHEREAS, the Stock Purchase Agreement provides, among other
things, that Tiffany will have the right to subscribe to certain issuances of
the Company's capital stock (and rights to acquire such capital stock); and

                  WHEREAS, in order to induce the Stockholder to relinquish
certain of its rights to facilitate the transactions contemplated by the Stock
Purchase Agreement, the Company has agreed to grant the Stockholder the right to
subscribe to certain issuances of the Company's capital stock (and rights to
acquire such capital stock);

                  NOW, THEREFORE, in consideration of the foregoing, the parties
hereto agree as follows:

                  1.       DEFINITIONS.  For purposes of this Agreement:

                           1.1      "AFFILIATE" means, as to a Person, any other
Person that directly or indirectly, through one or more intermediaries controls,
is controlled by or is under common control with the first-mentioned Person.

                           1.2      "COMMON SHARE EQUIVALENTS" means Common
Stock at the time outstanding or issuable upon conversion of shares of Preferred
Stock at the time outstanding, or issuable upon exercise of warrants or options
to purchase Common Stock or upon conversion or exercise of any other security.

                           1.3      "COMMON STOCK" means shares of Common Stock,
par value $.01, of the Company.
<PAGE>

                           1.4      "PERSON" means an individual, corporation,
partnership, limited liability company, association, trust, unincorporated
organization or other legal entity.

                           1.5      "PREFERRED STOCK" means shares of Preferred
Stock, par value $.01, of the Company.

                           1.6      "STOCKHOLDER" means Jewelcor Management,
Inc., its Affiliates and, for purposes of Section 2 of this Agreement, shall
include Seymour Holtzman, his siblings, spouse, lineal descendants and any
trusts for the benefit of the foregoing; PROVIDED, HOWEVER, that the foregoing
definition shall not be deemed to constitute an admission that the foregoing
persons are includable in any "group" as such term is defined in Rule 13d-3
promulgated under Section 13 of the Securities Exchange Act of 1934, as amended.
"Stockholder" shall not include Marc Holtzman, Steven Holtzman or Allison
Holtzman Garcia.

         2.       SUBSCRIPTION RIGHTS.

                  2.1 The Company shall not issue, sell or exchange, agree to
issue, sell or exchange, or reserve or set aside for issuance, sale or exchange,
(i) any Common Stock, (ii) any other equity securities of the Company,
including, without limitation, shares of Preferred Stock, (iii) any option,
warrant or other right to subscribe for, purchase or otherwise acquire any
equity securities of the Company, or (iv) any debt or other securities directly
or indirectly convertible into capital stock of the Company (collectively, the
"Offered Securities"), unless in each such case the Company shall have first
complied with this Section 2. The Company shall deliver to the Stockholder a
written notice of any proposed or intended issuance, sale or exchange of Offered
Securities (the "Offer"), which Offer shall (i) identify and describe the
Offered Securities, (ii) describe the price and other terms upon which they are
to be issued, sold or exchanged, and the number or amount of the Offered
Securities to be issued, sold or exchanged, (iii) identify the persons or
entities (if known) to which or with which the Offered Securities are to be
offered, issued, sold or exchanged and (iv) offer to issue and sell to or
exchange with such Stockholder (A) a portion of the Offered Securities
determined by dividing the aggregate number of shares of Common Stock then held
by such Stockholder (giving effect to the conversion or exercise of all Common
Share Equivalents then held by such Stockholder) by the total number of shares
of Common Stock then outstanding (giving effect to the conversion or exercise of
all Common Share Equivalents then outstanding (other than those owned by such
Stockholder)) (the "Basic Amount"); PROVIDED, HOWEVER, that the Stockholder, in
its discretion, may subscribe for less than the Basic Amount.

                  2.2 To accept an Offer, in whole or in part, the Stockholder
must deliver a written notice to the Company prior to the end of the five (5)
day period following the Stockholder's receipt of the Offer, setting forth the
portion of the Stockholder's Basic Amount that such Stockholder elects to
purchase (the "Notice of Acceptance").

                  2.3 The Company shall have 90 days from the expiration of the
period set forth in Section 2.2 above to issue, sell or exchange all or any part
of such Offered Securities as to which a
<PAGE>

Notice of Acceptance has not been given by the Stockholder (the "Refused
Securities"), but only to the offerees described in the Offer (if so described
therein) and only upon terms and conditions (including, without limitation, unit
prices and interest rates) that are not more favorable to the acquiring person
or persons or less favorable to the Company than those set forth in the Offer.

                  2.4 In the event the Company shall propose to sell less than
all the Refused Securities (any such sale to be in the manner and on the terms
specified in Section 2.3 above), then the Stockholder may, at its sole option
and in its sole discretion, reduce the number or amount of the Offered
Securities specified in its Notice of Acceptance to an amount that shall be not
less than the number or amount of the Offered Securities that the Stockholder
elected to purchase pursuant to Section 2.2 above multiplied by a fraction, (i)
the numerator of which shall be the number or amount of Offered Securities the
Company actually proposes to issue, sell or exchange (including Offered
Securities to be issued or sold to the Stockholder pursuant to Section 2.2 above
prior to such reduction) and (ii) the denominator of which shall be the original
amount of the Offered Securities. In the event that the Stockholder so elects to
reduce the number or amount of Offered Securities specified in its Notice of
Acceptance, the Company may not issue, sell or exchange more than the reduced
number or amount of the Offered Securities unless and until such securities have
again been offered to the Stockholder in accordance with Section 2.1 above.

                  2.5 Upon the closing of the issuance, sale or exchange of all
or less than all of the Refused Securities, the Stockholder shall acquire from
the Company, and the Company shall issue to the Stockholder, the number or
amount of Offered Securities specified in the Notice of Acceptance, as reduced
pursuant to Section 2.4 above if the Stockholders have so elected, upon the
terms and conditions specified in the Offer. The purchase by the Stockholder of
any Offered Securities is subject in all cases to the preparation, execution and
delivery by the Company and the Stockholder of a purchase agreement relating to
such Offered Securities reasonably satisfactory in form and substance to the
Stockholder and its counsel.

                  2.6 Any Offered Securities not acquired by the Stockholder or
other persons in accordance with Section 2.3 above may not be issued, sold or
exchanged until they are again offered to the Stockholder under the procedures
specified in this Agreement.

                  2.7      The term "Offered Securities" shall NOT include:

                           (a)      Common Stock issued as a stock dividend to
holders of Common Stock or upon any subdivision or combination of Common Stock;
and

                           (b)      shares of Common Stock issued or issuable
pursuant to any Company employee stock option, employee purchase or similar
plan.

<PAGE>

         3.      BUSINESS OPPORTUNITIES.

                  3.1 In the event that (a) the Stockholder or any of its
Affiliates or (b) any officer, director or employee of the Company or any
subsidiary of the Company who is also an officer, director or employee of the
Stockholder or an Affiliate thereof, acquires knowledge of a potential
transaction which may be a business opportunity for both the Company and the
Stockholder or any of its Affiliates, such business opportunity shall belong to
the Stockholder and not to the Company, and any such officer, director of
employee of the Company shall treat such business opportunity as belonging only
to the Stockholder and not the Company, PROVIDED, HOWEVER, with respect to
clause (b) of this sentence, the Stockholder shall determine in good faith
whether, based on the circumstances under which such officer, director or
employee acquired his knowledge, such business opportunity was offered to such
person solely in his capacity as an officer, director or employee of the Company
("Company Capacity"). For the purposes of the foregoing determination, there
shall be a presumption that such opportunity was offered to such person in his
capacity as an officer, director or employee of the Stockholder or an Affiliate
thereof. In the event that the Stockholder determines that it was so offered to
such person in his Company Capacity, such business opportunity shall belong only
to the Company and not to the Stockholder and such officer, director or employee
shall treat such business opportunity as belonging only to the Company and not
to the Stockholder. With respect to any business opportunity belonging to the
Stockholder pursuant to this Section 3.1, the Stockholder shall decide how to
allocate and pursue such business opportunity based on its sole determination of
what is the b est interests of the Stockholder's stockholders. The good faith
determination of the allocation of business opportunities pursuant to the
Section 3.1 shall be conclusive and binding for all purposes.

         4.     MISCELLANEOUS.

                  4.1 SUCCESSORS AND ASSIGNS. The Stockholder may assign its
rights under this Agreement, in whole or in part, to any Affiliate, and the
terms and conditions of this Agreement shall inure to the benefit of and be
binding upon the respective successors and assigns of the parties. Except as
provided in the immediately preceding sentence, no party may assign its rights
or delegate its duties or obligations hereunder without the prior written
consent of the other party hereto. Nothing in this Agreement, express or
implied, is intended to confer upon any party other than the parties hereto or
their respective successors and assigns any rights, remedies, obligations, or
liabilities under or by reason of this Agreement, except as expressly provided
in this Agreement.

                  4.2     GOVERNING LAW; JURISDICTION; WAIVER OF JURY TRIAL.

                           (a)      This Agreement shall be governed by and
construed under the laws of the State of New York as applied to agreements among
New York residents entered into and to be performed entirely within New York.
<PAGE>

                           (b)      The jurisdiction and venue in any action
brought by any party hereto pursuant to this Agreement shall properly (but not
exclusively) lie in any federal or state court located in the State of New York.
By execution and delivery of this Agreement, each party hereto irrevocably
submits to the jurisdiction of such courts for himself or itself and in respect
of his or its property with respect to such action. The parties irrevocably
agree that venue would be proper in such court, and hereby waive any objection
that such court is an improper or inconvenient forum for the resolution of such
action. The parties further agree that the mailing by certified or registered
mail, return receipt requested, of any process required by any such court shall
constitute valid and lawful service of process against them, without necessity
for service by any other means provided by statute or rule of court.

                           (c)      BECAUSE DISPUTES ARISING IN CONNECTION WITH
COMPLEX FINANCIAL TRANSACTIONS ARE MOST QUICKLY AND ECONOMICALLY RESOLVED BY AN
EXPERIENCED AND EXPERT PERSON AND THE PARTIES WISH APPLICABLE LAWS TO APPLY
(RATHER THAN ARBITRATION RULES), THE PARTIES DESIRE THAT THEIR DISPUTES BE
RESOLVED BY A JUDGE APPLYING SUCH APPLICABLE LAWS. THEREFORE, TO ACHIEVE THE
BEST COMBINATION OF THE BENEFITS OF THE JUDICIAL SYSTEM AND OF ARBITRATION, THE
PARTIES HERETO WAIVE ALL RIGHT TO TRIAL BY JURY IN ANY ACTION, SUIT OR
PROCEEDING BROUGHT TO ENFORCE OR DEFEND ANY RIGHTS OR REMEDIES UNDER THIS
AGREEMENT OR ANY DOCUMENTS RELATED HERETO.

                  4.3 COUNTERPARTS; FACSIMILE SIGNATURES. This Agreement
may be executed in any number of counterparts, each of which shall be deemed to
be an original, and all of which together shall constitute one and the same
document. This Agreement may be executed by facsimile signatures.

                  4.4 TITLES AND SUBTITLES. The titles and subtitles used in
this Agreement are used for convenience only and are not to be considered in
construing or interpreting this Agreement.

                  4.5 NOTICES. Unless otherwise provided, any notice required
or permitted under this Agreement shall be given in writing and shall be deemed
effectively given upon receipt by the party to be notified or five (5) days
after deposit with the United States Post Office, by registered or certified
mail, postage prepaid and addressed to the party to be notified

         (a)      if to the Company, at the following address:

                  161-B Crown Bay
                  P.O. Box 930
                  St. Thomas, U.S. V. I.  00804
                  Attention:  Robert L. Baumgardner
                  Fax:  (340) 779-9900

<PAGE>

                  with a copy to:

                  Proskauer Rose LLP
                  1585 Broadway
                  New York, New York  10036-8299
                  Attn:  Jack P. Jackson, Esq.
                  Fax:  (212) 969-2900

         (b)      if to the Stockholder, at the address set forth opposite the
         Stockholder's name on the signature pages hereto

or at such other address as any of the parties may designate by ten (10) days'
advance written notice to the other parties.

                  4.6 AMENDMENTS AND WAIVERS. Any term of this Agreement may be
amended and the observance of any term of this Agreement (including, without
limitation, Section 3) may be waived (either generally or in a particular
instance and either retroactively or prospectively), only with the written
consent of the Company and the Stockholder.

                  4.7 TERMINATION. This Agreement shall terminate and be of no
further force on the first day that the Stockholder ceases to own at least 50%
of the number of shares of Common Stock owned by the Stockholder as of the date
such Stockholder became a party to this Agreement (as adjusted for any stock
splits, stock dividends or stock combinations).

                  4.8 SEVERABILITY. If one or more provisions of this Agreement
are held to be unenforceable under applicable law, such provision shall be
excluded from this Agreement and the balance of the Agreement shall be
interpreted as if such provision were so excluded and shall be enforceable in
accordance with its terms.

                  4.9 SPECIFIC PERFORMANCE. In addition to any and all other
remedies that may be available at law in the event of any breach of this
Agreement, the Stockholder shall be entitled to specific performance of the
agreements and obligations of the Company hereunder and to such other injunctive
or other equitable relief as may be granted by a court of competent
jurisdiction.

                  4.10 COMPLETE AGREEMENT. This Agreement constitutes the entire
agreement and understanding of the parties hereto with respect to the subject
matter hereof and supersedes all prior agreements and understandings relating to
such subject matter (including, without limitation, that certain letter of
intent, dated as of March 5, 2001, by and among the Company, the Stockholder,
Seymour Holtzman and Tiffany and Company) and negotiations and oral
understandings, if any, with respect thereto.


<PAGE>

                  4.11 PRONOUNS. Whenever the context may require, any pronouns
used in this Agreement shall include the corresponding masculine, feminine or
neuter forms, and the singular form of nouns and pronouns shall include the
plural, and vice versa.

                  IN WITNESS WHEREOF, the parties have executed this Investor's
Rights Agreement as of the date first above written.



                                          THE COMPANY:

                                          LITTLE SWITZERLAND, INC.


                                          By: /s/ Robert L. Baumgardner
                                             -----------------------------------
                                               Name:  Robert L. Baumgardner
                                               Title: President

                                          INVESTOR:

ADDRESS:                                  JEWELCOR MANAGEMENT, INC.

225 North East Mizner Boulevard
Boca Raton, FL  33432                     By: /s/ Seymour Holtzman
                                             -----------------------------------
Attention:  Seymour Holtzman                   Name:  Seymour Holtzman
Fax:  (561) 672-4758                           Title:    President