<PAGE>
                                                                    EXHIBIT 10.4

                       [Mayor's Jewelers, Inc. Letterhead]





                                                               February 20, 2004


Henry Birks & Sons Inc.
1240  Phillips Square
Montreal, Quebec H3B 3H4
Attn:  John Ball

         Re:      PAYMENT OF DIVIDENDS ON SERIES A-1 CONVERTIBLE PREFERRED STOCK
                  AND RELATED MATTERS

Dear Mr. Ball:

         This letter agreement confirms our agreement with respect to terms and
conditions applicable to the payment by Mayor's Jewelers, Inc., a Delaware
corporation (the "Company"), to Henry Birks & Sons Inc., a Canadian corporation
("Birks"), of dividends on the Company's Series A-1 Convertible Preferred Stock
(the "Preferred Stock"), such dividends are noted in the Certificate of
Designation of the Preferred Stock, dated February 20, 2004 (the "Certificate of
Designation"). Please verify Birks' acceptance of and agreement to the terms of
this Letter Agreement by countersigning this Letter Agreement below and
returning a fully executed copy to the Company, Attention Marc Weinstein.

         1.       PAYMENT OF DIVIDENDS.

                  Subject to the terms and conditions of this Letter Agreement,
and notwithstanding the Dividend Payment Dates set forth in the Certificate of
Designation (as such term is defined therein), the Company agrees to pay to
Birks the Initial Dividend (as such term is defined in the Certificate of
Designation) (the "Dividend Payment"), on February 20, 2004.

         2.       CONDITIONS PRECEDENT TO PAYMENT OF DIVIDENDS.

                  Birks acknowledges and agrees that, notwithstanding the
provisions of Paragraph 1 above, the Dividend Payment shall not be made by the
Company to Birks unless and until: (a) Regaluxe Investment Sarl, the controlling
stockholder of Birks, shall have granted a bridge loan to Birks in a principal
amount of not less than Two Million Five Hundred Thousand Canadian Dollars
(CAN$2,500,000) (the "Bridge Loan"); (b) GMAC shall have agreed to extend Birks
revolving credit facility on substantially the same terms for an additional 3
year period; (c) the Company shall have received (or, if this loan has not yet
been documented, will receive by February) from Birks fully executed copies of


                                       1
<PAGE>

the loan documents evidencing the loan described in (a) above, in form and
substance satisfactory to the Company; and (d) the Company shall have been
granted a loan from Back Bay Capital, LLC ("Back Bay") in a principal amount of
not less than Two Million U.S. Dollars (U.S.$2,000,000) (the "Company Loan") on
such terms and conditions as are satisfactory to the Board of Directors of the
Company.

         3.       REDUCTION OF DIVIDENDS.

                  a. Any and all fees incurred by the Company in connection with
the Dividend Payment and the Company Loan, including those fees set forth on
SCHEDULE I hereto, whether payable to the lenders or any other third party,
including, without limitation, all commitment fees, origination fees, investment
advisor fees, legal fees, and accounting fees, shall be immediately paid by
Birks upon receipt of the Dividend Payment; provided, however, to the extent
that a full reimbursement would result in net proceeds available to Birks of
less than CAN$2.5 million, the Company agrees to reduce future dividend payments
by the amount of any such deficit.

                  b. In addition to the Dividend Payment, beginning on February
28, 2005, Birks shall be entitled to receive, out of funds legally available
therefor, dividends on each share of Preferred Stock at a rate per annum equal
to $80 per share, as, when, and if declared by the Board of Directors.

                  c. If prior to February 28, 2005, the Company desires to
purchase or otherwise acquire (or is deemed to acquire in any business
combination or otherwise) any of the shares of the Preferred Stock, for the
shares of Preferred Stock acquired, the holder shall pay the Company a cash
payment equal to an amount calculated by multiplying (1) the number of shares
purchased by the Company by (2) a fraction, the numerator of which is the
product of (a) three thousand two hundred ninety eight dollars and sixty three
cents ($3,298.63) and (b) the number of days from the Event Date (as defined
herein) through and including February 28, 2005, and the denominator of which is
the number of shares of outstanding of Preferred Stock. Event Date is the date
of the purchase by the Company of the Preferred Stock.

                  d. (i) Except as provided in (ii) below, if a holder converts
or elects to convert any of its Preferred Stock on or prior to February 28,
2005, the conversion rate shall be calculated by multiplying 3,333.33 by the sum
of (a) one (1), minus (b) the Conversion Adjustment. "Conversion Adjustment"
shall be an amount, the numerator of which is three thousand two hundred ninety
eight dollars and sixty three cents ($3,298.63) multiplied by the number of days
from the date of conversion through and including February 28, 2005, and the
denominator of which is an amount equal to the product of (1) the number of
preferred shares then owned by the converting holder multiplied by 3333.33,
multiplied by (2) the average market price of the Company's stock during the
past 10 trading days.



                  For example, if Birks converted 15,050 shares of Preferred
         Stock on January 1, 2005, and the current market price was $0.85, the
         conversion rate would be calculated as follows:

                                                  (3,298.63 X 59)
                                                  ----------------
                  Conversion rate = 3333.33 X 1 - (15050 X 3333.33) X .85



                                       2
<PAGE>

                  Conversion rate  = 3333.33 X  (1 - .0046)

                  Conversion rate  = 3333.33 X .9954

                  Conversion rate  = 3318

                  (ii) Notwithstanding subsection (d)(i) above, a holder may
elect, on or prior to February 28, 2005, to convert all or part of the Preferred
Stock at a conversion rate of 1 share of Preferred Stock to 3,333.33 shares of
Common Stock; provided, such holder pays the Company for each share of Preferred
Stock converted, a cash payment equal to an amount, the numerator of which is
the product of (a) three thousand two hundred ninety eight dollars and sixty
three cents ($3,298.63) and (b) the number of days from the Conversion Date
through and including February 28, 2005, and the denominator of which is the
number of shares of outstanding Preferred Stock on the Conversion Date.
"Conversion Date" shall be defined as the date of conversion by the holders of
the Preferred Stock.


                  For example, if a holder converted 1,000 shares of Preferred
         Stock on January 1, 2005, and 15,050 shares of Preferred Stock were
         outstanding, the cash payment would be calculated as follows:

                                                              (3,298.63 X 59)
                                                              ----------------
                                    Cash Payment = 1,000 X        15,050

                                    Cash Payment = 1,000 X  12.93

                                    Cash Payment = $12,931.51

         4.       COVENANTS OF BIRKS.

                  a. Birks covenants and agrees that it shall use the proceeds
from the Dividend Payment solely for working capital purposes, and in accordance
with the projections provided by Birks to Capitalink, L.C. ("Capitalink") prior
to the date of this Letter Agreement. Birks further covenants and agrees that
Birks shall, from and after the date of this Letter Agreement, upon the
Company's request, execute, acknowledge and deliver such documents to cause the
satisfactory completion and consummation of the transactions contemplated by
this Letter Agreement.

                  b. Birks further covenants and agrees that it will not
authorize, declare, or make any dividend payment or other distribution to
Regaluxe Investment Sarl or any other shareholders prior to April 15, 2005;
provided, however, Birks shall be permitted to continue to make the following
payments to Regaluxe: (i) payment of consulting fees as have been historically
paid to Regaluxe, and (ii) repayment of principal and interest of approximately
$250,000 per year on behalf of HBS Holdings. In addition, upon the sale by Birks
of its own shares of stock, its shares of the Company's common stock or its


                                       3
<PAGE>

shares of the Preferred Stock and subject to the advance written approval of
GMAC, Birks shall be permitted to use the proceeds from such sale(s) to make the
following distributions: (x) repayment of principal and interest under the
Bridge Loan, and (y) repayment of a convertible debenture held by Regaluxe or an
affiliate in the amount of US$2.5 million.

         5.       REPRESENTATIONS AND WARRANTIES OF BIRKS.

                  Birks represents and warrants to the Company that:

                  (a) The execution and delivery of this Letter Agreement by
Birks and the consummation of the transactions contemplated hereby have been
duly and validly authorized by all necessary corporate action of Birks, and this
Letter Agreement is a legal, valid and binding obligation of Birks and is
enforceable against Birks in accordance with its terms, subject to the effect of
any applicable bankruptcy, insolvency, reorganization, moratorium or similar law
affecting creditors' rights generally and to general principles of equity.

                  (b) No statement, information, projection, report, document,
financial statement, certificate or instrument furnished by or on behalf of
Birks to the Company, the Company's advisors, including, without limitation,
counsel to the Company or Capitalink, or to the Company's Independent
Alternative Committee in connection with this Letter Agreement or the
transactions contemplated hereby, contains or will contain any untrue statement
of any fact, or omits or will omit to state any fact which is necessary in order
to make the information contained therein not misleading.

                  (c) The projections or forecasts provided by Birks to
Capitalink have been reasonably prepared on bases reflecting the best available
estimates and judgments of management at the time of their preparation, and the
projections and forecasts represent Birks management's best estimate of the
future results of its business operations based on market and other conditions
prevailing on February 20, 2004.

         6.       MISCELLANEOUS.

                  (a) This Letter Agreement contains the entire agreement
between the parties with respect to the subject matter hereof and supersedes all
prior agreements, understandings and negotiations, both written and oral,
between the parties with respect to the subject matter of this Letter Agreement.
This Letter Agreement shall be binding upon and inure to the benefit of the
permitted successors and assigns of each.

                  (b) This Letter Agreement shall be construed and interpreted
in accordance with the laws of the State of Florida, without regard to the
conflicts of law rules of such state. Jurisdiction and venue for all actions
regarding this Letter Agreement shall lie exclusively in state and federal
courts located in Broward County, Florida. The parties agree that should any
provision of this Letter Agreement require interpretation or construction that
all parties have participated in the drafting of this document and no
presumption regarding construing the document against one party shall apply.



                                       4
<PAGE>

                  (c) All waivers of the breach of any provision of this Letter
Agreement by any party shall be in writing, and one or more waivers of the
breach of any provision of this Letter Agreement by any party shall not be
construed as a waiver of a subsequent breach of the same or any other provision,
nor shall any delay or omission by a non-defaulting party to seek a remedy for
any breach of this Letter Agreement or to exercise the rights accruing to it
constitute a waiver of its remedies and rights with respect to such breach.

                  (d) No provision of this Letter Agreement may be amended
except in writing executed by the parties hereto. Neither party may assign this
Letter Agreement without the prior written consent of the other party.

         This Letter Agreement shall become effective upon execution by the
parties. This Letter Agreement may be executed in counterparts, each of which
shall be deemed an original, but all of which together shall constitute one and
the same instrument; provided that a facsimile signature shall be considered due
execution.

                               MAYOR'S JEWELERS, INC.



                               By: /s/ MARC WEINSTEIN
                                   ------------------------------------------
                               Name:  Marc Weinstein
                               Title: Sr. VP and Chief Administrative Officer

Accepted and agreed to this 20th day of February, 2004:


HENRY BIRKS & SONS INC.

By:    /s/ JOHN BALL              /s/ MARCO PASTERIS
       -------------              ------------------
Name:  John Ball                   Marco Pasteris
Title: CFO                         Group VP, Finance








                                       5