<PAGE>   1
                                                                    Exhibit 10.4

                  PROXY AND VOTING AGREEMENT, dated as of February 8, 2001 (this
"AGREEMENT"), among New Valley Corporation, a Delaware corporation ("NEW
VALLEY"), Ladenburg, Thalmann Group Inc., a Delaware corporation ("LTGI"),
Berliner Effektengesellschaft AG, a German corporation ("BERLINER") and the
individual stockholders of GBI Capital Management Corp. listed on SCHEDULE A
hereto (collectively, the "PRINCIPALS"). As used in this Agreement, the term
"Principal" means, with respect to each individual listed on SCHEDULE A hereto,
such individual and, where applicable, the Living Trust set forth below his
name.

                  WHEREAS, the Selling Parties, GBI Capital Management Corp., a
Florida corporation (the "PURCHASER"), and Ladenburg, Thalmann & Co. Inc., a
Delaware corporation ("LADENBURG"), are parties to a Stock Purchase Agreement
dated February 8, 2001 (the "STOCK PURCHASE AGREEMENT"; capitalized terms not
defined herein shall have the meanings ascribed to them in the Stock Purchase
Agreement), pursuant to which the Purchaser shall acquire all of the outstanding
shares of capital stock of Ladenburg from the Selling Parties;

                  WHEREAS, the Sellers have the right to acquire shares of
Purchaser Common Stock pursuant to the Stock Purchase Agreement; and

                  WHEREAS, the Selling Parties are entering into the Stock
Purchase Agreement in reliance upon the execution and delivery of this Agreement
by the Principals;

                  NOW, THEREFORE, in consideration of the mutual covenants and
agreements set forth in this Agreement, and for other good and valuable
consideration, the receipt and sufficiency of which are hereby acknowledged, the
parties hereto agree as follows:

SECTION 1. REPRESENTATIONS AND WARRANTIES. Each Principal represents and
warrants to the Selling Parties that:

                  (a) Such Principal owns, beneficially and of record, as of the
date hereof, the number of shares of Purchaser Common Stock set forth next to
his name in SCHEDULE A hereto (collectively, the "SHARES"), subject to no rights
of others and free and clear of all security interests, liens, claims, pledges,
options, rights of first refusal, agreements, limitations on the Principal's
voting rights, charges and other encumbrances of any nature whatsoever other
than those imposed by federal and state securities laws. On the date hereof, the
Shares constitute all of the shares of Purchaser Common Stock beneficially owned
by each such Principal. Except as provided in those certain agreements between
Frost-Nevada, Limited Partnership ("FROST-NEVADA") and each of Messrs.
Rosenstock, Mangone, Thalheim and Zeitchick, respectively (copies of which has
been provided to New Valley), to sell an aggregate of 550,000 shares of
Purchaser Common Stock to Frost-Nevada, such Principal's right to vote or
dispose of the Shares beneficially owned by such Principal is not subject to any
voting trust, voting agreement, voting arrangement or proxy and such Principal
has not entered into any contract, option or other arrangement or undertaking
with respect thereto.


<PAGE>   2


                  (b) Each Principal has the legal capacity to execute, deliver
and perform this Agreement. This Agreement constitutes a valid and binding
obligation of each Principal enforceable against him in accordance with its
terms. If each Principal is married and the Shares constitute community property
under applicable law, this Agreement has been duly authorized, executed and
delivered by, and constitutes the valid and binding agreement of, the
Principal's spouse enforceable against such spouse in accordance with its terms.

                  (c) The execution, delivery and performance by each Principal
of this Agreement and the consummation of the transactions contemplated hereby
do not and will not (i) result in any breach or violation of or be in conflict
with or constitute a default under the terms of any law, order, regulation or
agreement or arrangement to which he is a party or by which he is bound, (ii)
require any filing with or authorization by any governmental entity or (iii)
require any consent or other action by any person under, constitute a default
under, or give rise to any right of termination, cancellation or acceleration or
to a loss of any benefit to which he is entitled under any provision of any
agreement or other instrument binding on him.

SECTION 2. VOTING AGREEMENT. (a) Until the Closing Date, no Principal will
assign, sell, pledge, hypothecate or otherwise transfer or dispose of any of the
shares of Purchaser Common Stock beneficially owned by such Principal, or any
other securities of the Purchaser with respect to which he otherwise has the
right to vote, or any interest therein, deposit any of such shares or securities
into a voting trust or enter into a voting agreement or arrangement or grant any
proxy with respect thereto (except as contemplated by this Proxy and Voting
Agreement) or enter into any contract, option or other arrangement or
undertaking with respect to the direct or indirect transfer or disposition of
any of the shares or securities. In the case of any transfer by operation of
law, this Agreement shall be binding upon the transferee.

                  (b) Each Principal will, with respect to those shares of
Purchaser Common Stock or other securities of the Purchaser that such Principal
either owns for voting at the Purchaser Stockholder Meeting to be held for the
purpose of voting on the adoption of the Transaction Documents and the issuance
of shares of Purchaser Common Stock pursuant to the Transaction Documents or for
granting any written consent in connection with the solicitation of written
consents in lieu of such a meeting or with respect to which such Principal
otherwise controls the vote, vote or cause to be voted such shares (or execute
written consents with respect to such shares) (i) to approve the Transaction
Documents, the issuance of shares of Purchaser Common Stock pursuant to the
Transaction Documents and the transactions contemplated thereby, (ii) against
any Purchaser Alternative Transaction and (iii) in favor of any other matter
necessary for the consummation of the transactions contemplated by the
Transaction Documents.

                  (c) Each Principal acknowledges that concurrently with the
execution of this Agreement, such Principal has executed and delivered to LTGI
an Irrevocable Proxy, pursuant to Section 607.0722 of the Florida Business
Corporation Act, coupled with an interest, the form of which is attached hereto
as EXHIBIT A, so as to vote such shares in accordance with this SECTION 2 and
each Principal hereby grants to LTGI such irrevocable proxy. The terms of this
proxy shall expire upon approval by the requisite vote of the Purchaser's
stockholders at the Purchaser Stockholder Meeting or at any adjournment thereof




                                      -2-
<PAGE>   3


of the adoption of the Transaction Documents and issuance of the Purchaser
Common Stock as contemplated thereby or upon the earlier termination of the
Stock Purchase Agreement in accordance with the provisions thereof.

                  (d) The Principals, New Valley and LTGI shall use commercially
reasonable efforts to cause the agreements in this SECTION 2 to be appropriately
disclosed in filings with the Commission, including the Proxy Statement referred
to in the Stock Purchase Agreement.

SECTION 3. NO SOLICITATION. Prior to the Closing Date, no Principal shall, and
each Principal shall use best efforts to cause such Principal's Affiliates
(other than the Purchaser) and Representatives not to, initiate, solicit or
encourage, directly or indirectly, any inquiries or the making or implementation
of any Purchaser Alternative Transaction, or engage in any negotiations
concerning, or provide any confidential information or otherwise facilitate any
effort or attempt to make or implement, a Purchaser Alternative Transaction. The
Principals will promptly notify the Selling Parties if any such inquiries,
proposals or offers are received by, any such information is requested from, or
any such negotiations or discussions are sought to be initiated or continued
with, such Principal or any of such Persons. Notwithstanding the foregoing,
nothing in this Section shall prevent any Principal from taking the actions
referred to in Section 5.16 of the Stock Purchase Agreement, but only in the
circumstances and subject to the conditions specified therein.

SECTION 4. BINDING EFFECT. All covenants, representations, warranties and other
stipulations in this Agreement and other documents referred to herein, given by
or on behalf of any of the parties hereto, shall bind and inure to the benefit
of the respective successors, heirs, personal representatives and assigns of the
parties hereto.

SECTION 5. ENTIRE AGREEMENT. This Agreement contains the entire agreement among
the parties with respect to the subject matter hereof and supersedes all prior
arrangements or understandings with respect hereto.

SECTION 6. NOTICES. All notices, requests, consents and other communications
hereunder to any party shall be deemed to be sufficient if contained in a
written instrument and shall be deemed to have been duly given when delivered in
person, by telecopy, by nationally-recognized overnight courier, or by first
class registered or certified mail, postage prepaid, addressed to such party at
the address set forth below or such other address as may hereafter be designated
in writing by the addressee to the addressor at the address and telecopier
numbers set forth in the Stock Purchase Agreement, with respect to the Selling
Parties, and at the addresses and telecopier numbers set forth in SCHEDULE A for
the Principals. All such notices, requests, consents and other communications
shall be deemed to have been delivered when received.

SECTION 7. MODIFICATIONS; AMENDMENTS; WAIVERS. The terms and provisions of this
Agreement may not be modified or amended, nor any provision hereof waived,
except pursuant to a writing signed by the parties hereto (including their
assigns). No waiver by any party of any term of this Agreement in any one or
more instances shall be deemed or construed as a waiver of such term on any
future occasion.




                                      -3-
<PAGE>   4



SECTION 8. COUNTERPARTS. This Agreement may be executed in any number of
counterparts, and each such counterpart hereof shall be deemed to be an original
instrument, but all such counterparts together shall constitute but one
agreement.

SECTION 9. HEADINGS. The headings of the various sections of this Agreement have
been inserted for convenience of reference only and shall not be deemed to be a
part of this Agreement.

SECTION 10. SEVERABILITY. It is the desire and intent of the parties that the
provisions of this Agreement be enforced to the fullest extent permissible under
the law and public policies applied in each jurisdiction in which enforcement is
sought. Accordingly, if any provision of this Agreement would be held in any
jurisdiction to be invalid, prohibited or unenforceable for any reason, such
provision, as to such jurisdiction, shall be ineffective, without invalidating
the remaining provisions of this Agreement or affecting the validity or
enforceability of such provision in any other jurisdiction. Notwithstanding the
foregoing, if such provision could be more narrowly drawn so as not to be
invalid, prohibited or unenforceable in such jurisdiction, it shall, as to such
jurisdiction, be so narrowly drawn, without invalidating the remaining
provisions of this Agreement or affecting the validity or enforceability of such
provision in any other jurisdiction.

SECTION 11. GOVERNING LAW. This Agreement shall be governed by and construed in
accordance with the law of the State of New York, without giving effect to
principles governing conflicts of laws, except to the extent that provisions of
the Florida Business Corporation Act apply, which provisions the parties cannot
legally waive or otherwise exclude, exempt or release themselves from by
contract.

SECTION 12. SPECIFIC PERFORMANCE: REMEDIES. The parties hereto agree that
irreparable damage would occur in the event any provision of this Agreement was
not performed in accordance with the terms hereof and that the parties shall be
entitled to specific performance of the terms hereof, in addition to any other
remedy at law or equity. Except as otherwise expressly provided for herein, no
remedy conferred by any of the specific provisions of this Agreement is intended
to be exclusive of any other remedy, and each and every remedy shall be
cumulative and shall be in addition to every other remedy given hereunder or now
or hereafter existing at law or in equity or by statute or otherwise. The
election of any one or more remedies by any party hereto shall not constitute a
waiver by any such party of the right to pursue any other available remedies.

SECTION 13. CONSENT TO JURISDICTION; SERVICE OF PROCESS. Each of the parties
hereto hereby irrevocably submits to the exclusive jurisdiction of the courts of
the State of New York and to the jurisdiction of the United States District
Court for the Southern District of New York or any court of the State of New
York located in the Borough of Manhattan in the City of New York, for the
purpose of any action or proceeding arising out of or relating to this Agreement
and each of the parties hereto hereby irrevocably agrees that all claims in
respect to such action or proceeding shall be heard and determined exclusively
in any New York state or federal court. Each of the parties hereto agrees that a




                                      -4-
<PAGE>   5


final judgment in any action or proceeding shall be conclusive and may be
enforced in other jurisdictions by suit on the judgment or in any other manner
provided by law. Each of the parties hereto irrevocably consents to the service
of the summons and complaint and any other process in any other action or
proceeding relating to the transactions contemplated by this Agreement, on
behalf of itself or its property, by personal delivery of copies of such process
to such party. Nothing in this Section 13 shall affect the right of any party to
serve legal process in any other manner permitted by law.

SECTION 14. WAIVER OF JURY TRIAL. EACH OF THE PARTIES HERETO HEREBY IRREVOCABLY
WAIVES ALL RIGHT TO TRIAL BY JURY IN ANY ACTION, PROCEEDING OR COUNTERCLAIM
(WHETHER BASED ON CONTRACT, TORT OR OTHERWISE) ARISING OUT OF OR RELATING TO
THIS AGREEMENT OR THE ACTIONS OF THE PARTIES HERETO IN THE NEGOTIATION,
ADMINISTRATION, PERFORMANCE AND ENFORCEMENT THEREOF.

SECTION 15. FURTHER ASSURANCES. Each party will take such further actions as may
reasonably be requested by another party to effect the purposes of this
Agreement.

SECTION 16. TERMINATION. This Agreement shall terminate upon a termination of
the Stock Purchase Agreement in accordance with the provisions of its terms.

                      [SIGNATURE PAGE IMMEDIATELY FOLLOWS]


                                      -5-
<PAGE>   6


         IN WITNESS WHEREOF, the parties hereto have executed this Proxy and
Voting Agreement on the date first written above.

                               NEW VALLEY CORPORATION

                               By:  /s/ RICHARD J. LAMPEN
                                  -----------------------------------------
                                    Name:   Richard J. Lampen
                                    Title:  Executive Vice President

                               LADENBURG, THALMANN GROUP INC.

                               By:  /s/ VICTOR M. RIVAS
                                  -----------------------------------------
                                    Name:   Victor M. Rivas
                                    Title:  Chairman & Chief Executive Officer


                               BERLINER EFFEKTENGESELLSCHAFT AG

                               By:  /s/ HOLGER TIMM
                                  -----------------------------------------
                                    Name:   Holger Timm
                                    Title:  Chief Executive Officer

                               PRINCIPALS:

                                   /s/ JOSEPH BERLAND
                                  -----------------------------------------
                                 JOSEPH BERLAND


                               JOSEPH BERLAND REVOCABLE LIVING TRUST
                               DTD 4/16/97


                               By:  /s/ JOSEPH BERLAND
                                  -----------------------------------------
                                     Title: Trustee



                                      -6-
<PAGE>   7




                         /s/ RICHARD J. ROSENSTOCK
                        -----------------------------------------
                         RICHARD J. ROSENSTOCK

                         RICHARD J. ROSENSTOCK REVOCABLE LIVING TRUST
                         DTD 3/5/96


                         By:  /s/ RICHARD J. ROSENSTOCK
                        -----------------------------------------
                               Name:  Richard J. Rosenstock
                               Title: Trustee


                           /s/ MARK ZEITCHICK
                        -----------------------------------------
                         MARK ZEITCHICK

                           /s/ VINCENT A. MANGONE
                        -----------------------------------------
                         VINCENT A. MANGONE


                         VINCENT A. MANGONE REVOCABLE LIVING TRUST
                         DTD 11/5/96


                         By:  /s/ VINCENT A. MANGONE
                        -----------------------------------------
                               Name:  Vincent A. Mangone
                               Title: Trustee

                           /s/ DAVID THALHEIM
                        -----------------------------------------
                         DAVID THALHEIM


                         DAVID THALHEIM REVOCABLE LIVING TRUST
                         DTD 3/5/96


                         By:  /s/ DAVID THALHEIM
                        -----------------------------------------
                               Name:  David Thalheim
                               Title: Trustee




                                      -7-
<PAGE>   8

                                   SCHEDULE A
<TABLE>
<CAPTION>

                  NAME, ADDRESS AND FAX NUMBER                                 NUMBER OF SHARES
                  ----------------------------                                 ----------------

<S>                                                                                <C>
                         JOSEPH BERLAND                                            3,945,060
       Joseph Berland Revocable Living Trust dated 4/16/97

                c/o GBI Capital Management Corp.
                       1055 Stewart Avenue
                       Bethpage, NY 11714

                  Facsimile No.: (516) 470-1050

                      RICHARD J. ROSENSTOCK                                        3,945,060
    Richard J. Rosenstock Revocable Living Trust dated 3/5/96

                c/o GBI Capital Management Corp.
                       1055 Stewart Avenue
                       Bethpage, NY 11714

                  Facsimile No.: (516) 470-1050

                         MARK ZEITCHICK                                            1,512,273

                c/o GBI Capital Management Corp.
                       1055 Stewart Avenue
                       Bethpage, NY 11714

                  Facsimile No.: (516) 470-1050

                       VINCENT A. MANGONE                                          1,512,273
     Vincent A. Mangone Revocable Living Trust dated 11/5/96

                c/o GBI Capital Management Corp.
                       1055 Stewart Avenue
                       Bethpage, NY 11714

                  Facsimile No.: (516) 470-1050

                         DAVID THALHEIM                                            1,512,273

       David Thalheim Revocable Living Trust dated 3/5/96

                c/o GBI Capital Management Corp.
                       1055 Stewart Avenue
                       Bethpage, NY 11714

                  Facsimile No.: (516) 470-1050

</TABLE>