<PAGE>
                                                                    EXHIBIT 10.4

                       CO-SALE AND VOTING RIGHTS AGREEMENT

         THIS CO-SALE AND VOTING RIGHTS AGREEMENT (this "Agreement") is made and
entered into as of the 11th day of April, 2003, by and between Marshall T. Leeds
("Leeds") and Antares Capital Fund III Limited Partnership, a Delaware limited
partnership (the "Purchaser" or "Antares").

         WHEREAS, pursuant to that certain Stock Purchase Agreement entered into
by and between Summit Brokerage Services, Inc., a Florida corporation (the
"Company"), and the Purchaser as of the date hereof (the "Purchase Agreement"),
it was and is a condition precedent and covenant of the Company to cause this
Agreement to be entered into, executed and delivered to the Company by Leeds;

         WHEREAS, Leeds has agreed to grant the Purchaser the opportunity to
participate, upon the terms and conditions set forth in this Agreement, in
subsequent sales of the Common Stock (as defined in the Purchase Agreement) made
by any Leeds Affiliate (as hereinafter defined); and

         WHEREAS, Leeds has agreed to vote all of his shares of Common Stock
with respect to certain matters as set forth within Section 4 herein, as may be
directed by the Purchaser.

         NOW, THEREFORE, in consideration of the foregoing premises, the mutual
covenants and agreements set forth herein and other good and valuable
consideration, the sufficiency of which is hereby acknowledged, the parties
hereto hereby agree as follows:


                                    ARTICLE 1
                            SALES BY LEEDS AFFILIATE

         1.1 TRANSFERS. Any sale, transfer or other disposition, whether
voluntarily or by operation of law or otherwise, in any transaction or series of
related transactions, of any shares of Common Stock (each a "Transfer") by (i)
Leeds, (ii) each of his family members and (iii) each entity (other than the
Company) which is directly or indirectly majority-owned by Leeds or any of his
family members or for which Leeds or any of his family members or an entity
directly or indirectly majority-owned by Leeds or any of his family members
serves as the sole or managing general partner or the sole or managing member
(the foregoing items (i), (ii) and (iii) and each and all combinations thereof,
are collectively referred to herein as the "Leeds Affiliates" and each
individually as a "Leeds Affiliate"), shall be void, of no force or effect, and
transfer no right, title or interest in or to any such Common Stock shares to
the purported transferee, except to the extent that the applicable Leeds
Affiliate(s) comply(ies) in all respects with the terms and conditions of this
Agreement. For purposes hereof a family member of Leeds is his spouse, children
or any other lineal descendents.

         1.2 RIGHT TO PARTICIPATE. During the Applicable Period (as such term is
defined in the Purchase Agreement), if a Leeds Affiliate desires to effectuate a
Transfer other than to a Leeds Affiliate, then, at least fifteen (15) business
days prior to the closing of such Transfer, Leeds shall, by written notice,
offer (the "Participation Offer") to Antares the right (but not the obligation)
to participate (the "Tag Along Rights"), exercisable by giving written notice of



                                       1
<PAGE>

exercise to Leeds within seven (7) business days after Antares' receipt of the
Participation Offer, to sell in the Transfer that number of Common Stock shares
as shall be determined pursuant to Section 1.2(a) below at the same price and
upon the same terms and conditions and otherwise treated the same, in substance,
as those applicable to the proposed Transfer of the Common Stock by the Leeds
Affiliate (the "Transferee Terms"). The Participation Offer shall set forth, in
reasonable detail, each and all of the Transferee Terms (including, but not
limited to, the proposed Transferee, purchase price, payment terms, closing date
and any and all other material terms). Each and all actual or proposed Transfers
by a Leeds Affiliate, other than a Permitted Transfer pursuant to and in
accordance with Section 1.5 below, shall be subject to such Tag Along Rights,
and shall be conditioned upon all applicable Leeds Affiliates complying in full
with the provisions of this Agreement. For purposes of this Agreement, the
Purchaser shall be deemed to include, where applicable and without limitation,
the Purchaser and any Affiliates (as such term is defined in Rule 405 of the
Securities Act of 1933, as amended (the "Securities Act") thereof, other than
the Company, including, but not limited to, its partners (and their assignees)
and their respective Affiliates to whom Common Stock shares were/are
distributed. The Tag Along Rights of the Purchaser shall be subject to the
following terms and conditions:

                  (a) The Purchaser may sell in the Transfer all or any part of
that number of shares of Common Stock equal to the product obtained by
multiplying (i) the aggregate number of shares of Common Stock covered by the
proposed Transfer by (ii) a fraction the numerator of which is the number of
shares of Common Stock then owned by or on behalf of the Purchaser (including,
but not limited to, its partners or Affiliates other than the Company) and the
denominator of which is the sum of the number of shares of Common Stock then
owned by all Leeds Affiliates (including shares Transferred to Permitted
Transferees as hereinafter defined in accordance herewith) and the Purchaser
(including, but not limited to, its partners or Affiliates other than the
Company). For purposes of making such computation, the number of shares of
Common Stock deemed to be owned by the Purchaser and its Affiliates and the
Leeds Affiliates shall include any and all exercisable and in-the-money options,
warrants and other Common Stock equivalents.

                  (b) The Purchaser may participate in the proposed Transfer by
delivering to Leeds, together with the notice of exercise described above in
this Section 1.2, one or more certificates, properly endorsed for Transfer to
the proposed Transferee or its designee (the "Transferee"), which represent the
number of shares of Common Stock which the Purchaser elects to sell pursuant to
this Section 1.2. In addition, the Purchaser shall execute and deliver to
Transferee a stock sale/purchase agreement as may be reasonably required by the
Transferee, in such form and with such representations and warranties regarding
the title to and encumbrances on the Purchaser's Common Stock so endorsed,
Purchaser's authority to transfer the Common Stock, and other terms that are
customary for an investor of the same character as the Purchaser at the time of
the transfer who is selling to a third party common stock in a public company of
the Company's size and type and which are reasonably acceptable to Purchaser;
provided, however, that Purchaser's failure to execute and deliver to Transferee
any such agreement shall not impair the Transfer of Common Stock by any Leeds
Affiliate pursuant to the Transferee Terms, and in such event Leeds shall return
to Purchaser its endorsed certificates and shall have no further obligation to
Purchaser under this Agreement with respect to the subject Transfer or any sale
proceeds received by any Leeds Affiliate in connection therewith.




                                       2
<PAGE>

         1.3 CONSUMMATION OF SALE. Subject to Section 1.2(b) hereof, the stock
certificate or certificates which the Purchaser delivers to Leeds pursuant to
Section 1.2(b) shall be delivered by Leeds to the proposed Transferee against
payment of the appropriate purchase price or other consideration therefor as
provided in the Participation Offer and subject to and not before the
consummation of the sale of the Common Stock pursuant to the Transferee Terms
specified in the Participation Offer, including, without limitation, the outside
date of the closing thereof, and Leeds shall promptly, but in no event later
than the date of the closing of the proposed Transfer, cause to be paid to the
Purchaser by the Transferee that portion of the sale proceeds to which the
Purchaser is entitled by reason of its exercise of Tag Along Rights with respect
to such sale.

         1.4 ONGOING RIGHTS. The exercise or non-exercise of the Tag Along
Rights of the Purchaser shall not adversely affect its rights to participate in
subsequent Common Stock Transfers by Leeds pursuant to Sections 1.1 and 1.2
hereof.

         1.5 PERMITTED EXEMPTIONS. The Tag Along Rights of the Purchaser shall
not apply to (a) any pledge of Common Stock made by a Leeds Affiliate pursuant
to a bona fide loan transaction which creates a mere security interest, or (b)
any gift by pledge or transfer to a charitable organization, or (c) any Transfer
to a Leeds Affiliate; PROVIDED, HOWEVER, that (i) Leeds shall provide prior
written notice to Purchaser of such pledge or Transfer at least five (5)
calendar days prior to effecting it and (ii) the pledgee or Transferee thereof
(each, a "Permitted Transferee") shall furnish the Purchaser with a written
agreement to be bound by and comply with all provisions of this Agreement
applicable to Leeds.

         1.6 DEFINITIONS. For purposes of this Agreement, the term "Applicable
Period" shall have the meaning ascribed to such term in the Purchase Agreement.


                                    ARTICLE 2
                              PROHIBITED TRANSFERS

         2.1 TREATMENT OF PROHIBITED TRANSFERS. In the event that,
notwithstanding the provisions of Section 1 hereof, a Leeds Affiliate should
effectuate an effective Transfer of any Common Stock in contravention and breach
of the Tag Along Rights of the Purchaser (a "Prohibited Transfer"), then the
Purchaser, in addition to such other remedies as may be available at law, in
equity or hereunder (including, without limitation, enforcing the voidability of
the Prohibited Transfer as provided in Section 1 hereof), shall have the put
option provided in Section 2.2 below, and Leeds shall be bound by the applicable
provisions of such put option.

         2.2 PUT OPTION. In the event of a Prohibited Transfer, the Purchaser
shall have the right, and one available remedy for such breach shall be, to sell
to Leeds, and Leeds shall have the obligation to purchase from the Purchaser, a
number of shares of Common Stock equal to the number of shares the Purchaser
would have been entitled to sell to the purchaser in the Prohibited Transfer
pursuant to the terms of Section 1 hereof. Such sale shall be made on the
following terms and conditions:

                  (a) The price per share (the "Share Price") at which the
shares are to be sold to Leeds shall be the sum of (i) the Share Price paid by
the purchaser to the Leeds Affiliate in the Prohibited Transfer and (ii) simple
interest on the Share Price, computed at a rate equal to eighteen percent (18%)



                                       3
<PAGE>

per annum, pro rated for the period of time between the payment in full of the
purchase price with respect to the put option described herein by Leeds and
receipt by the Leeds Affiliate of any proceeds from the Prohibited Transfer
giving rise to the put option.

                  (b) Within 90 days after the later of the dates on which the
Purchaser (i) received notice from Leeds of the Prohibited Transfer or (ii)
otherwise becomes aware of the Prohibited Transfer, the Purchaser shall, if
exercising the put option created hereby, deliver to Leeds the certificate or
certificates representing the Purchaser's Common Stock shares to be sold, each
certificate to be properly endorsed for Transfer.

                  (c) Leeds shall, concurrently with its receipt of the
certificate or certificates for the shares to be sold by the Purchaser pursuant
to Section 2.2(b), pay and deliver to the Purchaser in cash the aggregate
purchase price therefor as specified in Section 2.2(a), by certified check or
bank draft made payable to the order of Purchaser.


                                    ARTICLE 3
                              LEGENDED CERTIFICATES

         3.1 LEGEND. Each certificate representing shares of Common Stock now or
hereafter owned by any Leeds Affiliate or issued to any Permitted Transferee
pursuant to Section 1.5 shall be endorsed with the following legend:



                  "RESTRICTIONS ON THE RIGHT TO SELL OR TRANSFER THE SHARES OF
         STOCK REPRESENTED BY THIS CERTIFICATE HAVE BEEN IMPOSED PURSUANT TO A
         CO-SALE AND VOTING RIGHTS AGREEMENT BETWEEN THE HOLDER OF THIS
         CERTIFICATE AND ANTARES CAPITAL FUND III LIMITED PARTNERSHIP, A COPY OF
         WHICH IS ON FILE WITH THE OFFICE OF, AND AVAILABLE UPON WRITTEN REQUEST
         TO, THE SECRETARY OF THE COMPANY."



         3.2 LEGEND REMOVAL. The Section 3.1 legend shall be removed upon
termination of this Agreement in accordance with the provisions concerning
termination set forth herein and therein.

                                    ARTICLE 4
                        SHARE VOTING AND OTHER AGREEMENTS

         4.1 BOARD OF DIRECTORS. During the Applicable Period, Leeds agrees
that, if so directed by the Purchaser, he shall take and cause to be taken such
shareholder actions and/or vote and cause to be voted all shares of Common Stock
beneficially held by each Leeds Affiliate (including, but not limited to, any
and all shares of Common Stock and Common Stock equivalents owned directly or
indirectly by Mr. Richard Parker or Affiliates thereof over which Leeds has
voting control) in favor of each of the following: (i) the nomination (if
necessary) and election to the Company's board of directors of a designee



                                       4
<PAGE>

selected by the Purchaser from time to time in its sole discretion (the
"Purchaser Director"), (ii) the maintenance of the Purchaser Director as a
member of the Company's board of directors at all times during the Applicable
Period, and (iii) to the extent a vote or other action is required to be taken
by the shareholders of the Company, or such shareholders (by vote or other
action) could effectuate same, (A) the appointment of the Purchaser Director to
the compensation and audit committees of the board of the directors of the
Company and the maintenance of such Purchaser Director as a member of such
committees at all times during the Applicable Period, (B) the maintenance at all
times of at least two (2) Independent Directors (as such term is defined in the
Purchase Agreement) on the Company's board of directors, and (C) a requirement
that the membership of each of the audit and compensation committees of the
Company's board of directors be limited exclusively to Independent Directors and
at least two (2) thereof.

         4.2 STOCK ISSUANCES. During the Applicable Period, Leeds agrees that he
shall not put before the Company's board of directors or any committee thereof
for its consideration, nor shall he accept if so voted upon by the Company's
board of directors or any committee thereof, the issuance (the "Issuance") of
any equity securities of the Company or securities convertible, exchangeable
and/or exercisable thereinto, including, without limitation, any Common Stock,
preferred stock, options, warrants, and other equity equivalents (collectively,
"Securities") to any Leeds Affiliate if the price to be paid for such Securities
is below $.25 per share (or the equivalent thereof after giving effect to any
splits, mergers, stock dividends, recapitalizations and similar transactions
effectuated after the Closing Date (as such term is defined in the Purchase
Agreement)); provided, however, that the term Issuance shall not be deemed to
include the Company's issuance of Securities to a Leeds Affiliate at a price
below $.25 per share (or the equivalent thereof after giving effect to any
splits, mergers, stock dividends, recapitalizations and similar transactions
effectuated after the Closing Date), subject to and only with the approval of a
majority of the Company's Independent Directors, under either of the following
circumstances: (A) Common Stock, options and/or warrants, but in no event
preferred stock, as compensation for services in lieu of cash, but only to the
extent that (i) the value placed on such Common Stock and the exercise price per
share for such options or warrants are not, at the time of issuance, below the
greater of (1) the last closing sale price of the Common Stock on the trading
day nearest the issuance date and (2) the average of the closing sale prices for
the Common Stock for the 20 trading days immediately preceding the issuance
date, and (ii) any and all such issuances (Common Stock, options and/or
warrants) in the aggregate do not exceed 5% of the then outstanding shares of
Common Stock; or (B) in connection with the purchase of Securities (but in no
event preferred stock) by a Leeds Affiliate, the proceeds of which are necessary
for the Company's continued operations, the completion of a merger or an
acquisition (excluding exercise by Leeds of his currently outstanding options),
or such other appropriate business purpose as is approved by at least a majority
of the board of directors; provided, however, that in the case of the
immediately preceding item (B), the Purchaser will have the right to participate
in such an investment with the applicable Leeds Affiliates to the extent of the
product obtained by multiplying (i) the aggregate number of Securities covered
by the proposed issuance by (ii) a fraction the NUMERATOR of which is the sum of
the number of shares of Common Stock and all options, warrants and other Common
Stock equivalents (whether or not such options, warrants and other Common Stock
equivalents are then "in the money" or exercisable) then owned by or on behalf
of the Purchaser (including, but not limited to, its partners or Affiliates
other than the Company), and the DENOMINATOR of which is the sum of the number
of shares of Common Stock and all options, warrants and other Common Stock



                                       5
<PAGE>

equivalents (whether or not such options, warrants and other Common Stock
equivalents are then "in the money" or exercisable) then owned by or on behalf
of the Leeds Affiliates and the Purchaser (including, but not limited to, its
partners or Affiliates other than the Company).

         4.3 RELATED-PARTY TRANSACTIONS. Leeds hereby agrees and covenants that,
during the Applicable Period, no Leeds Affiliate will enter, directly or
indirectly, into a joint venture, loan or any other transaction, agreement or
arrangement with the Company without first obtaining the consent of the majority
of the Company's Independent Directors.

         4.4 EMPLOYMENT AGREEMENT. During the Applicable Period, Leeds agrees
not to effectuate or otherwise directly or indirectly cause, and not to propose,
any amendments or other modifications to Leeds' amended and restated employment
agreement with the Company dated May 22, 2002 without first obtaining the
consent of the majority of the compensation committee of the Company's board of
directors, including, without limitation, any salary and/or benefit increases
necessary and/or appropriate to make Leeds' salary commensurate with industry
standards.

                                    ARTICLE 5
                            MISCELLANEOUS PROVISIONS

         5.1 TERMINATION OF CO-SALE AND VOTING RIGHTS AGREEMENT. The rights of
the Purchaser under this Agreement and the obligations of Leeds and the other
Leeds Affiliates with respect to the Purchaser shall terminate upon the
expiration of the Applicable Period. Unless sooner terminated in accordance with
the preceding sentence, this Agreement shall terminate upon the closing of a
Qualified Public Offering of the Common Stock. Upon termination of this
Agreement, Leeds shall, and shall cause each of the other Leeds Affiliates to,
surrender to the Company (or its successor) their stock certificates and the
Company (or its successor) shall issue to such Leeds Affiliates an equal number
of Common Stock shares (or successor securities thereto) without the legend set
forth in this Agreement. For purposes of this Section 5.1, a "Qualified Public
Offering" is defined as the initial public offering of shares of Common Stock
pursuant to an effective registration statement under the Securities Act
resulting in at least $20,000,000 of net proceeds to the Company at a price to
the public of at least $2.50 per share (subject to appropriate adjustment for
stock splits, stock dividends, recapitalizations and other similar events).

         5.2 NOTICES. Whenever notice is provided for in this Agreement, it
shall be given in writing and hand delivered, or mailed by registered or
certified mail, return receipt requested, or sent by facsimile to the party or
parties to whom addressed at the addresses or facsimile numbers set forth below:
The date of delivery shall be the date received if delivered by hand or sent by
facsimile with written confirmation received, or within three (3) days of
mailing, if mailed. Any party may change the address to which notice shall be
delivered or mailed by notice given pursuant to this Section 5.2.

         If to the Purchaser:      Antares Capital Fund III Limited Partnership
                                   7900 Miami Lakes Drive West
                                   Miami Lakes, Florida 33016
                                   Attn: Mr. Jonathan I. Kislak
                                   Facsimile:  (305) 894-3227



                                       6
<PAGE>

         With copy to:             Bilzin Sumberg Baena Price & Axelrod LLP
                                   2500 Wachovia Financial Center
                                   200 South Biscayne Boulevard
                                   Miami, FL  33131-5340
                                   Attn: Alan D. Axelrod, Esq.
                                   Facsimile: (305) 374-7593

         If to Leeds:              Marshall T. Leeds
                                   Summit Brokerage Services, Inc.
                                   980 North Federal Highway
                                   Suite 310
                                   Boca Raton, Florida 33432
                                   Facsimile:  (561) 347-6705

         With copy to:             Greenberg Traurig, P.A.
                                   450 S. Orange Avenue, Suite 650
                                   Orlando, Florida 32801
                                   Attn:  Sandra C. Gordon, Esq.
                                   Facsimile:  (407) 420-5909

         5.3 SUCCESSORS AND ASSIGNS. This Agreement and the rights and
obligations of the parties hereunder shall inure to the benefit of, and be
binding upon, their respective successors, assigns and legal representatives.
The Tag Along Rights of the Purchaser hereunder are only assignable (i) by the
Purchaser to any partner of Purchaser or to an Affiliate of Purchaser or any of
its partners, or (ii) to an assignee or transferee who acquires the Purchaser's
Common Stock shares other than pursuant to Rule 144 or pursuant to a
registration statement declared effective under the Securities Act.

         5.4 SEVERABILITY. In the event one or more of the provisions of this
Agreement should, for any reason, be held to be invalid, illegal or
unenforceable in any respect, such invalidity, illegality or unenforceability
shall not affect any other provisions of this Agreement, and this Agreement
shall be construed as if such invalid, illegal or unenforceable provision had
never been contained herein.

         5.5 AMENDMENTS. Any amendment or modification of this Agreement shall
be effective only if evidenced by a written instrument executed by duly
authorized representatives of the parties hereto. Any waiver by a party of its
rights hereunder shall be effective only if evidenced by a written instrument
executed by a duly authorized representative of such party. In no event shall
such waiver of any rights hereunder constitute the waiver of such rights in any
future instance unless the waiver so specifies in writing.

         5.6 GOVERNING LAW/JURISDICTION. This Agreement will be governed by,
construed and enforced in accordance with, the laws of the State of Florida.
Each of the parties hereto irrevocably consents to the exclusive jurisdiction
and venue of any court located within Miami-Dade or Palm Beach County, in the
State of Florida in connection with any matter based upon or arising out of this
Agreement or the matters contemplated herein, agrees that process may be served



                                       7
<PAGE>

upon them in any manner authorized by the laws of the State of Florida for such
persons and waives and covenants not to assert or plead any objection which they
might otherwise have to such jurisdiction, venue and such process.

         5.7 ATTORNEYS' FEES. The prevailing party in any dispute with respect
to this Agreement shall be entitled to recover from the other party all of its
reasonable costs and expenses incurred in connection with such dispute,
including, but not limited to, reasonable attorneys', paralegals', accountant's
and other professionals' fees and costs incurred before and at trial, at any
other proceeding, at all appellate levels and whether or not suit or any other
proceeding is brought.



                     [SIGNATURES LOCATED ON THE NEXT PAGE.]




                                       8
<PAGE>


         IN WITNESS WHEREOF, the parties hereto have duly executed this Co-Sale
and Voting Rights Agreement as of the day and the year first set forth above.

LEEDS:

MARSHALL T. LEEDS


By:      /s/ Marshall T. Leeds
         --------------------------------------------
         Marshall T. Leeds


THE PURCHASER:

ANTARES CAPITAL FUND III LIMITED PARTNERSHIP,
a Delaware limited partnership


By:      ANTARES CAPITAL PARTNERS III, L.L.C.,
         a Florida limited liability company,
         its General Partner


By:      /s/ Jonathan I. Kislak
         --------------------------------------------
         Jonathan I. Kislak,
         Class A Member


                                       9