<PAGE>   1
                UNITED STATES SECURITIES AND EXCHANGE COMMISSION
                             WASHINGTON, D.C. 20549

                                    FORM 10-K

                  ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d)
                     OF THE SECURITIES EXCHANGE ACT OF 1934

                      FOR THE YEAR ENDED DECEMBER 31, 1996

                         COMMISSION FILE NUMBER 0-25882

                                VIDEOSERVER, INC.
             (Exact name of registrant as specified in its charter)

              DELAWARE                                    04-3114212
(State or other jurisdiction of incorporation  (IRS Employer Identification No.)
or organization)

                63 THIRD AVENUE, BURLINGTON, MASSACHUSETTS 01803
          (Address of principal executive offices, including Zip Code)

                                 (617) 229-2000
              (Registrant's telephone number, including area code)

           Securities Registered Pursuant to Section 12(b) of the Act:
                                      None

           Securities Registered Pursuant to Section 12(g) of the Act:
                           Common Stock $.01 par value

         Indicate by check mark whether the registrant (1) has filed all reports
required to be filed by Section 13 or 15(d) of the Securities Exchange Act of
1934 during the preceding 12 months (or for such shorter period that the
registrant was required to file such reports), and (2) has been subject to such
filing requirements for the past 90 days. Yes [x] No[ ]

         Indicate by check mark if disclosure of delinquent filers pursuant to
Item 405 of Regulation S-K is not contained herein, and will not be contained,
to the best of registrant's knowledge, in definitive proxy or information
statements incorporated by reference in Part III of this Form 10-K or any
amendment to this Form 10-K. [ ]

         The aggregate market value of voting stock held by non-affiliates of
the registrant as of March 7, 1997 was $290,511,143 (based on the last reported
sale price on the Nasdaq National Market on that date).

         The number of shares outstanding of the registrant's Common Stock as of
March 7, 1997 was 12,641,004.

                       DOCUMENTS INCORPORATED BY REFERENCE

 1.  Portions of the definitive Proxy Statement to be delivered to Shareholders
     in connection with the Annual Meeting of Shareholders to be held May 14,
     1997 are incorporated by reference herein.
 2.  Portions of the 1996 Annual Report to Shareholders are incorporated by
     reference herein.
 3.  Portions of the registrant's Registration Statement on Form S-1
     (Registration No. 33-91132) are incorporated by reference herein.
<PAGE>   2
                                VIDEOSERVER, INC.
                          1996 FORM 10-K ANNUAL REPORT
                                TABLE OF CONTENTS

<TABLE>
<CAPTION>
                                                                                              PAGE
                                                                                              ----
<S>               <C>                                                                         <C>
                                     PART I

Item 1.            Business ...............................................................      3

Item 2.            Description of Property ................................................     13

Item 3.            Legal Proceedings ......................................................     13

Item 4.            Submission of Matters to a Vote of Security Holders ....................     13


                                     PART II

Item 5.            Market for Registrant's Common Equity and Related Stockholder Matters...     14
Item 6.            Selected Financial Data ................................................     14

Item 7.            Management's Discussion and Analysis of Financial Condition and Results
                   of Operations ..........................................................     14

Item 8.            Financial Statements and Supplementary Data ............................     14

Item 9.            Changes in and Disagreements with Accountants
                   on Accounting and Financial Disclosure .................................     14


                                    PART III

Item 10.           Directors and Executive Officers of the Registrant .....................     15

Item 11.           Executive Compensation .................................................     15

Item 12.           Security Ownership of Certain Beneficial Owners and Management .........     15
Item 13.           Certain Relationships and Related Transactions .........................     15


                                     PART IV

Item 14.           Exhibits, Financial Statements Schedules and Reports on Form 8-K .......     16

Signatures ................................................................................     18
</TABLE>

    This Report contains "forward-looking statements" within the meaning of the
Private Securities Litigation Reform Act of 1995. Such statements are subject to
certain risks and uncertainties, including without limitation those discussed in
the Company's 1996 Annual Report to Shareholders in the section titled "Other
factors which may affect future operations" (which section is hereby
incorporated by reference herein). Such forward-looking statements speak only as
of the date on which they are made, and the Company cautions readers not to
place undue reliance on such statements.

                                       2
<PAGE>   3
                                     PART I

ITEM 1.  BUSINESS

    VideoServer, Inc. ("VideoServer" or "the Company") is a leading supplier of
networking equipment and associated software used to create multimedia
conferences that connect multiple users over wide area networks and allow them
to interact as a group. The Company's products, Multimedia Conference Servers
(MCSs), provide multipoint conferencing, as well as applications for conference
control, network management and bandwidth management. VideoServer's products
enable interoperability between dissimilar communications networks, video
conferencing systems and the encoding technologies used by those systems. The
Company's products, which combine audio, video and data information, are
positioned to become a key enabler of communications networks supporting
emerging collaborative multimedia applications.

    VideoServer's objective is to expand its leadership position in the
conferencing market by developing and supplying products that enable multipoint
conferencing, whether through end-user multimedia networks or carrier-provided
conferencing services.

    The Company provides Multimedia Conference Servers to a wide variety of
technology and distribution partners, including leading videoconferencing
equipment suppliers, telephone carriers, private conferencing service providers,
computer companies and PBX suppliers. The Company believes that it was the
largest supplier in annual shipments of multimedia conference servers in 1994,
1995, and 1996.

    In March 1997, the Company entered into a definitive purchase agreement with
Promptus Communications to acquire certain assets comprising Promptus' network
access card business, for approximately $21,000,000 in cash and stock. The
transaction is expected to close in the second calendar quarter of 1997.
Promptus manufactures and markets a comprehensive line of digital network access
products used for a variety of applications, including videoconferencing, ISDN
Internet access, remote LAN access, and high-speed file transfer.  Network
access technology is an important component of the Company's MCS products, and
Promptus is a significant supplier to the Company.

INDUSTRY BACKGROUND

VIDEOCONFERENCING

    The multimedia conferencing market has evolved from videoconferencing. In
the late 1980s, with increasing numbers of conferencing endpoints installed and
customers desiring to connect multiple locations into the same conference,
videoconferencing equipment suppliers introduced multipoint control units (MCUs)
to transfer video and audio signals between all conference participants.
However, because of the proprietary nature of the encoding used, video terminals
and associated multipoint control units from different manufacturers were not
interoperable. In December 1990, the International Telecommunications Union
(ITU) introduced the H.320 standards for videoconferencing over switched digital
circuit networks to provide a framework for equipment from different
manufacturers to communicate with each other. Compatibility is particularly
important for communication via these networks, since the advantage of these
services is dial-up communications without regard to the type of equipment being
used at the receiving ends of the transmission.

    In recent years, lower-priced, standards-based products emerged that have
expanded the market for videoconferencing systems. Room systems that were once
priced in the $100,000 to $200,000 range are now priced as low as $8,000. In
addition, increasing competition between network carriers, and increasing demand
for applications such as Internet access, are leading to wider availability and
significantly declining costs of digital lines.

    With the introduction of chip sets incorporating ITU standards, a growing
number of companies have entered the desktop videoconferencing market, either
with stand-alone complete PC-based systems or with board sets that plug into
personal computers. Companies shipping H.320 compliant desktop conferencing
products include

                                       3
<PAGE>   4
British Telecom, Compression Labs, IBM, Intel, Olivetti, PictureTel, VTEL and
VCON. The costs of such desktop conferencing products have rapidly declined with
the introduction of more powerful chip sets and further miniaturization of
components such as cameras and monitors. Since early 1994, desktop conferencing
subsystems have dropped in price from $6,000 per seat to less than $1,500, with
some products available for as low as $1,000. As with the decrease in the price
of room systems, this decrease in price has fueled demand for desktop endpoints.

    Extending the reach of conferencing, in November 1995 the ITU introduced the
H.324 standard for low bit rate video, enabling videoconferencing over analog
lines using PCs or videophones. This is expected to bring conferencing
capability to an entirely new segment of end users, including those in smaller
businesses and homes. PC vendors such as AST, Compaq, Hewlett-Packard, IBM,
Sony, and Toshiba, and modem suppliers such as Boca Research and U.S. Robotics
began offering H.324 conferencing capabilities late in 1996.

    In May 1996, another important expansion of conferencing standards was
realized with the introduction of the H.323 standard governing real-time
collaboration over local area networks, intranets, and the Internet. Enabling
conferencing over traditional business networks provides a foundation for the
adoption of this application as a mainstream business tool. Vendors such as
Intel, PictureTel, and Microsoft have announced their intention to introduce
H.323 based endpoints in 1997.

    Each of these endpoints, whether room systems or desktop systems, require
server resources to conduct a multipoint conference. A larger installed base of
endpoints is expected to lead to more multipoint conferencing, and therefore to
a greater demand for the Company's MCS products.

COLLABORATIVE MULTIMEDIA APPLICATIONS

    Concurrent with advances in videoconferencing, significant investments have
been made in software to extend traditional desktop computing applications into
conference enabled real-time information sharing tools. Collaborative data
conferencing applications are emerging that redefine the way groups can work
together. With the ability to see and hear one another over telecommunications
lines and share a common desktop application like a white board, spreadsheet or
word-processing document, participants can share ideas and collaborate in real
time to improve the work product.

    In response to the emerging customer demand for multimedia applications, in
February 1996 the ITU extended the T.120 standards for collaborative multimedia
conferencing wherein video, audio and data information can be shared between
endpoints in a multipoint setting. In 1996 Microsoft bundled a standards-based
realtime voice and data conferencing package, NetMeeting, into its Windows
operating system. This product will provide an embedded collaborative capability
for potentially millions of PC users. In addition, Intel and IBM are offering
collaborative desktop applications with optional audio and video capabilities
sold through retail computer channels. Fundamental to the architecture of these
products is the presence of a multimedia conference server to provide the
multipoint link and data distribution mechanism among all the endpoints.

CARRIER-PROVIDED SERVICES

    The intense competition among carriers has increased the demand for
technologies that allow carriers to provide additional value-added services.
Multimedia conferencing technology offers such an opportunity, and carriers are
initiating collaborative multimedia conferencing services that provide on-demand
multipoint conferencing capability allowing users to connect their terminals to
multimedia conference servers located in the carriers' central offices.


                                       4
<PAGE>   5
A NEW CLASS OF NETWORK EQUIPMENT

    These trends -- the growth in room systems and desktop videoconferencing,
growth in collaborative desktop conferencing and expansion in services offered
by carriers -- are driving the need for sophisticated, networked multimedia
conference servers.

THE VIDEOSERVER SOLUTION

    VideoServer was founded in 1991 to develop a new generation of networking
equipment architected for the videoconferencing market as well as the emerging
multimedia conferencing market. The VideoServer MCS product line is built on an
industry standard hardware and software platform that combines a powerful set of
real-time conferencing applications with management tools and network
connectivity features that address today's customer requirements and are
positioned to meet emerging requirements. These requirements include:

    INTEROPERABILITY. VideoServer's Multimedia Conference Servers today provide
transparent interoperability among many different kinds of endpoints such as
room videoconferencing systems, desktop video terminals, and regular telephones
in the same conference, using combinations of voice, video, and data. In the
future, this level of interworking will be expected to encompass emerging
conferencing endpoint technologies such as data conferencing computers,
audio-graphics systems and videophones. Similarly, interoperability must be
provided between the many different brands of equipment and applications. The
technology also must be able to accommodate various encoding algorithms used to
compress multimedia information. The Company has expended substantial effort to
make its MCS interoperable with the products of virtually all suppliers of
standards-based videoconferencing terminals. In many cases this has required
subtle accommodations in the MCS for the specific characteristics of each brand
of endpoint due to the manufacturer's implementation of the ITU standards and
product performance. When different encoding technology is employed in terminals
for audio algorithms, the MCS provides the transcoding needed to deliver the
audio mix to each endpoint in the proper algorithm for that endpoint.

    CONNECTIVITY. Conference servers must be able to provide gateways between
diverse network services and between multiple carriers. While today's servers
provide connectivity largely to T-1, ISDN, private digital networks, and analog
voice networks, in the future, endpoints are expected to be connected to
ethernet and token ring local area networks, the Internet, and to advanced
networks based on ATM. In addition, as carriers add features to their networks,
conference servers must be able to support them.

    FUNCTIONALITY. In addition to video switching and audio mixing, application
features are needed to facilitate ease of operation, perform centralized
processing, interconnect with traditional data network servers and deliver many
new kinds of network information. These network servers will be required to
scale throughout the enterprise, provide redundancy for high reliability and
incorporate network management capabilities.

PRODUCTS

    The Company provides technology advances to its OEM and carrier customers
through products that incorporate rapid standards deployment, extensive feature
content, scalability, network flexibility and interoperability. The Company
believes that its technology leadership enables its customers to reduce their
time-to-market and their own product costs, and that the Company's technology
relationships with customers allow it to anticipate market requirements critical
to its products.

    The Company's products have been designed within a scaleable, modular
architecture to allow the customer to add capacity, processing power and
conferencing features as the customer's network and application requirements
grow. Using a common set of hardware and software building blocks, customers can
choose from a wide range of product configurations that differ in capacity,
price, network connectivity and features, all of which


                                       5
<PAGE>   6
share the same operating software user interface. The product may be configured
for use in customer premises environments or may be configured with specialized
packaging for use in a telephone carrier's central office setting.

    The MCS product line includes a number of basic platform configurations that
are expanded by the customer's selection of optional processing modules and
software applications. The platforms, configured for the typical end-user, range
in list price from under $25,000 to more than $200,000. Each MCS configuration
is built from a common set of processing modules, network interfaces, software
systems and optional features.

    The following table lists the basic chassis configurations offered by the
Company and the typical target market and application in which each is used. In
this table, user capacity is a measure of the number of simultaneous conference
participants that can be connected to the MCS.

<TABLE>
<CAPTION>
         MODEL    CAPACITY                        TARGET MARKET/APPLICATION
         -----    --------                        -------------------------
         <S>      <C>            <C>
         2007       8 users      Entry level Audio/Video/Data (A/V/D) multipoint for customer premises
                                 equipment (CPE) networks; distributed server
         2012       16 users     Mid-range CPE network central server
         2020       48 users     Large CPE/central office network with extensive multimedia applications
         CO         48 users     High availability central office server
</TABLE>

    Each of these systems may be interconnected to provide support for larger
conferences.

    The VideoServer MCS has an extensive number of available software and
hardware features, some of which are listed in the following table.
<TABLE>
<CAPTION>
                     APPLICATIONS                                      DESCRIPTION
                     ------------                                      -----------
         <S>                                    <C>

         CONFERENCE  SERVICES AND MANAGEMENT
         Continuous Presence ..............     Continuous viewing of multiple conference sites
         Multimedia Conferencing.........       Simultaneous audio visual conferencing and data conferencing
         Reservation and Scheduling.........    Schedule and manage MCS use
         Directory Services.................    Database of potential conference participants and sites
         Chairperson Conference Control         Management of conference activities by a selected video
                                                participant
         Security and Password Control......    Conference password and application security controls
         Accounting and Billing.............    System usage tracking for service billing
         Voice Activated Switching..........    Dynamic switching of video presentation based on current
                                                speaker
         Audio Add-on.......................    Conferencing module for audio-only conference participants
         Operator Attended Conferencing         Operator interface at conference initiation
</TABLE>


                                       6
<PAGE>   7
<TABLE>
<CAPTION>
         NETWORK SERVICES AND
         MANAGEMENT
<S>                                            <C>
         Centralized Multipoint Management
                                               Multi-MCS conference and network management package
         Outbound Dialing.................     Optional feature for MCS dial-out capability
         Conference Monitor...............     Real-time monitor of conference activities and status
         Bandwidth Management.............     Bandwidth aggregation using inverse multiplexing
         Event Management.................     System activity and alarms applications for network management
         Network Diagnostics..............     Network loop-back and problem isolation tool kit
         Premise Switching...............      Adds the function of an ISDN switch
</TABLE>

    The entry level Model 2007 MCS, continuous presence, T.120 conferencing
capability, operator attended conferencing features and premise switching all
began shipping in 1996.

    The VideoServer MCS can be directly connected to public networks (either T-1
or ISDN networks, or both) or private networks. The T-1 interface can be
configured as either a full or fractional T-1 (FT-1). If FT-1 service is
selected, multiple FT-1 circuits may be multiplexed and delivered by the network
to the MCS in a single T-1 pipe. ISDN Primary Rate Interface (PRI) support
allows the MCS to cost-effectively support multiple basic rate terminal
connections across a single interface. ISDN Basic Rate Interface (BRI) support
offers a cost effective solution for customer premise applications not requiring
PRI. The MCS supports the various ISDN network protocols used in the United
States, the United Kingdom, France, Germany, Japan, Australia and the
European-wide standard, and thus can be used worldwide.

    The Company offers add-on software to its installed base in the form of
either major new software releases or unbundled software options. Customers may
purchase new software releases on an as-needed basis or as part of a maintenance
agreement. Unbundled software options are priced separately and are not part of
maintenance agreements.

MARKETS AND CUSTOMERS

    VideoServer markets its products to Original Equipment Manufacturers (OEMs),
which generally resell the Company's products to end-users, and to service
providers, including public and private telephone carriers, which generally
offer conferencing services based on the Company's products. This distribution
strategy focuses on four types of customers: conferencing equipment
manufacturers and resellers, telephone companies as conferencing service
providers, computer systems and workstation companies, and PBX and network
equipment suppliers.

    Traditional videoconferencing equipment suppliers have historically
represented the primary market for delivering conferencing equipment to users.
The Company has relationships with most of the significant videoconferencing
suppliers around the world, including Compression Labs, Inc. (CLI) and
PictureTel Corporation in the United States, and British Telecom, GPT Video
Systems (a division of the UK-based GPT Limited), and Societe Anonyme de
Telecommunications (a division of the French-based Sagem Group), three of the
leading European suppliers of conferencing products. Each is offering the
Company's products to its customers under private label. The Company also has
agreements with leading Japanese manufacturers, including Sony Corporation,
which resells the products globally, as well as NEC America Inc., Fujitsu
Business Communications Systems, Inc. and Panasonic Corporation, which remarket
the products primarily in English-speaking countries worldwide.


                                       7
<PAGE>   8
    Telecommunication service providers are increasingly seeking to
differentiate themselves by offering multimedia conferencing services to
customers who desire on-demand conferencing capability without installing their
own conference servers. The Company markets its products and services directly
to public and private telecommunications service providers, including
Interexchange Carriers (IXCs) such as AT&T, MCI and Sprint, Regional Bell
Operating Companies (RBOCs) such as BellSouth and Southwestern Bell, a number of
private conferencing service providers in the U.S. such as Link VTC and
1-800-Video On, and to international Post Telephone and Telegraph companies
(PTTs) in Europe and Asia, such as British Telecom, Deutsche Telekom, France
Telecom, and NTT.

    As conferencing moves to desktop computing, computer systems companies are
beginning to address the emerging collaborative multimedia market. The Company
believes these companies view multimedia applications as a strategic technology
thrust that will fuel demand for computing resources and network bandwidth.
Since late 1993, Intel Corporation and VideoServer have participated in joint
development and marketing of products for the conferencing market. In late 1996,
the Company and Compaq Computer announced their intention to work closely with
Compaq's value-added resellers in North America to incorporate VideoServer
technology into Compaq servers.

    With the continued expansion of standards, network equipment and PBX
companies, whose products form the data and telecommunications backbone of the
enterprise network, are beginning to incorporate conferencing technology into
their product lines. In late 1996, the Company announced an OEM arrangement with
Cisco Systems, initially under which the Company will provide standards-based
conferencing gateways to Cisco for integration into certain of its enterprise
routers. Additionally, major PBX companies such as Nortel and Siemens/Rolm, who
are expanding their traditional audio-based offerings to include video and data
capabilities, have begun to resell the Company's MCS products.

    The Company's agreements with its customers generally do not include minimum
purchase requirements and are non-exclusive. In 1994, CLI and British Telecom
accounted for 55% and 12% of net sales. In 1995, CLI, PictureTel, who became a
customer during 1995, and BellSouth accounted for 23%, 19% and 18% of net sales.
In 1996, PictureTel and CLI accounted for 43% and 10% of net sales. Sales to
international customers accounted for 29%, 25% and 32% of the Company's net
sales for the years ended December 31, 1994, 1995 and 1996.

    VideoServer conducts its sales and marketing activities from its principal
offices in Burlington, Massachusetts, as well as from four other North American
sales offices and its European headquarters located in the United Kingdom.

RESEARCH AND PRODUCT DEVELOPMENT

    The Company believes that its future success depends on its ability to
continue to enhance and expand its existing products and to develop new products
which maintain its technology leadership. VideoServer has invested, and expects
to continue to invest heavily, in the development of products and core
technologies. Extensive product development input is obtained from OEM partners,
from service providers, from end users, and through the Company's active
participation and leadership in industry groups responsible for establishing
technical standards such as the ITU and the IMTC .

    Since its founding, the Company's research and development effort has been
directed towards the development of standards-based conference server
technology. In concert with the evolution of industry standards, these efforts
currently are focused on extending the breadth of network services supported
beyond switched digital services to include local area networks, corporate
intranets, the Internet, and ATM. This includes the development of multipoint
products for particular network platforms and gateway products to provide
interoperability between dissimilar network types. Development is also underway
to support emerging data conferencing applications, provide additional
conferencing management capabilities including enhanced user interfaces, and to
add higher capacities to the product family. The Company is extending and
accelerating its



                                       8
<PAGE>   9
efforts through development relationships with its customers. The Company
receives funding under certain of these arrangements which, when earned, is
recorded as a reduction of research and development expense in the Company's
financial statements.

    At December 31, 1996, VideoServer's research and development staff consisted
of 77 employees, including 65 software engineers and 12 hardware engineers. The
Company's net research and development expenditures were $3.3 million, $5.3
million and $7.8 million in 1994, 1995 and 1996, representing 21%, 19% and 16%
of net sales in those years. Development funding from customers of $340,000,
$1,000,000 and $1,150,000 was recorded as a reduction of research and
development costs in 1994, 1995 and 1996. All software development costs have
been expensed as incurred because costs eligible for capitalization have not
been material to date.

    In January 1997, the Company began to establish an international development
operation in the United Kingdom.

CUSTOMER SUPPORT AND SERVICE

    The Company provides technical support and services to its resellers and
customers. A high level of continuing service and support is critical to the
Company's objective of developing long-term relationships with customers. The
Company's resellers install, maintain and provide on-site and headquarters-level
technical support of products to their end-user customers, and VideoServer
provides comprehensive problem management, training, diagnostic tools, repair
services and spare parts to facilitate and supplement these efforts. The Company
installs, maintains and directly supports products sold to its direct customers.

    The Company offers a technical support hotline to its resellers and
customers. Network support engineers answer technical support calls placed by
the support engineers of the Company's resellers and by its direct customers.
The engineers generally provide same-day responses to questions that cannot be
resolved during the initial call. The products are designed with advanced remote
diagnostic capabilities that permit a reseller's or the Company's support
engineers to immediately begin the process of diagnosing any problems in the
field, thereby reducing both response time and cost. When necessary, however,
support engineers are dispatched to the customer's facility.

    The Company warrants its software products for 90 days. During this 90-day
warranty period, the Company will investigate all reported problems and will
follow escalation procedures to provide resolution. The Company warrants its
hardware products for 12 months. During this warranty period, the Company will
repair or replace any failed hardware component. The Company also offers
post-warranty support programs ranging from services on a time-and-materials
basis to full-service contracts on a 24-hour, 7-days-a-week basis, and a full
suite of training courses.

MANUFACTURING

    The Company's manufacturing operations consist primarily of materials
planning and procurement, quality control of materials, components and
subassemblies and final product configuration and testing. The Company designs
the significant hardware subassemblies for its products and uses independent
third-party contract assembly companies to perform printed circuit board
assembly. The Company configures and tests the hardware and software in
combinations to meet a wide variety of customer requirements. The Company uses
automated testing equipment, "burn-in" procedures and comprehensive inspection
and statistical process control testing intended to assure the quality and
reliability of its products. The Company has received the International Standard
Organization (ISO) 9002 certification for quality.


                                       9
<PAGE>   10
         Although the Company generally uses standard parts and components for
its products, certain components, including key digital signal processors from
Texas Instruments, are presently available only from single sources or from
limited sources. The Company has no supply commitments from its vendors,
including Texas Instruments, and generally purchases components on a purchase
order basis as opposed to entering into long term procurement agreements with
vendors. The Company has been able to obtain adequate supplies of components in
a timely manner from current vendors, or when necessary to meet production
needs, from alternate vendors. The Company believes that, for the Texas
Instruments digital signal processors in particular, alternative sources of
supply would be difficult to develop over a short period of time and an
interruption in supply or a significant increase in the price of these
components would adversely affect the Company's operating results and business.

    Because of the generally short cycle between order and shipment and because
the majority of the Company's sales in each quarter results from orders booked
in that quarter, the Company does not believe that its backlog as of any
particular date is indicative of future sales levels.

COMPETITION

    The market for communications products is intensely competitive and is
subject to rapid technological change. Although to date the Company has
experienced limited competition from products with comparable capabilities, the
Company expects competition to increase significantly in the future. Currently,
the Company's principal competition from producers of multipoint control units
comes from Lucent Technologies, and to a lesser extent from VTEL Corporation.
Other companies such as Multilink, Teleos (a division of Madge Networks) Accord,
Databeam, and Outreach, have introduced or announced their intention to
introduce products which could be competitive with the Company's products.
Furthermore, PictureTel Corporation, currently the Company's largest customer,
and Radvision have announced their intention to introduce products related to
the new H.323 standards which may be competitive with products the Company
currently has under development, and others are expected to enter the H.323
conferencing market. This additional competition could adversely affect the
Company's sales and profitability through price reductions and loss of market
share. In particular, should one or more of the Company's current customers,
including videoconferencing equipment suppliers, telecommunications carriers or
traditional network equipment vendors choose to provide or distribute
competitive products (including their own products) and services, the Company's
business could be materially adversely affected. Many of the Company's current
and potential competitors have substantially greater financial, marketing and
technical resources than the Company.

    The principal competitive factors in the market for multimedia conference
servers are, and are expected to continue to be, breadth of network services
supported, conformance to industry standards, price per port, performance,
network management capabilities, transcoding capabilities, reliability and
customer support. While the Company believes it presently competes favorably in
all of these areas, there can be no assurance that it will continue to do so.

PROPRIETARY RIGHTS

    While the Company has applied for several patents, it currently holds only
one U.S. patent relating to its existing products, and relies on a combination
of contractual rights, trade secrets and copyright laws to establish and protect
its intellectual property rights. The Company believes that, because of the
rapid pace of technological change in the data communications and
telecommunications industries, the intellectual property protection for its
products is a less significant factor in the Company's success than the
knowledge, abilities and experience of the Company's employees, the frequency of
its product enhancements, its relationships with its partners, the effectiveness
of its marketing activities, and the timeliness and quality of its support
services.


                                       10
<PAGE>   11
         The Company is subject to the risk of adverse claims and litigation
alleging infringement of the intellectual property rights of others. In December
1994, the Company settled a patent infringement litigation brought against it by
Datapoint Corporation (Datapoint) for a cash payment by the Company of $500,000.
There can be no assurance that additional third parties will not assert claims
against the Company in the future with respect to the Company's current or
future products or that any such claims would not require the Company to enter
into license arrangements or result in protracted and costly litigation,
regardless of the merits of such claims. No assurance can be given that the
Company would prevail with respect to any such claim, or that a license to
third-party rights, if needed, would be available on acceptable terms.

    Patent infringement litigation still exists between Datapoint and two of the
Company's largest customers. In addition, Datapoint has written inquiry letters
to a significant number of others in the videoconferencing market offering to
sell them nonexclusive licenses under certain Datapoint patents in the
videoconferencing field (the Datapoint Patents). While the validity or scope of
the Datapoint Patents has not been adjudicated by a court, Datapoint has, in
effect, asserted that the Datapoint Patents cover certain aspects of multipoint
videoconferencing operations involving terminals and multipoint control units,
including the Company's MCSs. As a result of the December 1994 settlement, the
Company obtained a nonexclusive license for its MCS under the Datapoint Patents,
which license includes limited rights for the products and services of the
Company's customers. However, the conferencing market in general, and the
Company's future sales and operating results in particular, could be adversely
affected as a result of ongoing uncertainties regarding the Datapoint Patents.
Such uncertainty, and any related potential impact, is likely to exist until the
validity of the patents is adjudicated.

EMPLOYEES

    At December 31, 1996, the Company employed a total of 193 persons, including
77 in research and development, 73 in sales, marketing and customer support, 24
in manufacturing and 19 in finance and administration. Thirteen of the Company's
employees were located in the United Kingdom and the remainder were located in
the United States. None of the Company's employees are represented by a labor
organization and the Company believes that its relations with employees are
good.

    Competition for qualified personnel in the computer networking and
communications industry is intense. VideoServer believes that its future success
will depend on its continued ability to attract and retain qualified personnel.

EXECUTIVE OFFICERS OF THE REGISTRANT

    The executive officers of the Company are:

<TABLE>
<CAPTION>
    NAME                                    AGE      POSITION
    ----                                    ---      --------
<S>                                         <C>      <C>
    Robert L. Castle. . . . . . . . . .     47       President, Chief Executive Officer and Director
    Jules L. DeVigne. . . . . . . . .       57       Vice President of Worldwide Sales
    Rubin Gruber. . . . . . . . . . . .     52       Vice President of Business Development, and Director
    Derek M. James. . . . . . . . . .       54       Chief Technology Officer
    John Jones. . . . . . . . . . . . . .   49       Vice President of Customer Service and Support
    Walter A. Jones. . . . . . . . . . .    49       Vice President of Research and Development
    Stephen J. Nill. . . . . . . . . . .    45       Vice President of Finance and Chief Financial Officer, Treasurer,
                                                     and Secretary
    John E. O'Neil. . . . . . . . . . .     41       Vice President of Marketing
    Frank T. Winiarski. . . . . . . .       54       Vice President of Manufacturing
</TABLE>

--------------------

                                       11
<PAGE>   12
    Robert L. Castle has served as President and Chief Executive Officer of the
Company since March 1993, and as a Director since March 1992. From February 1992
until March 1993, he served as President and Chief Operating Officer. Prior to
joining the Company, Mr. Castle was employed for eight years at FileNet
Corporation, a supplier of document imaging equipment, in various positions
including Senior Vice President of Marketing from October 1990 to February 1992
and Vice President of Marketing from December 1987 to October 1990. Previously,
Mr. Castle held marketing and general management positions at Basic Four Corp.,
a developer of software applications, and Sycor, Inc., a developer and
manufacturer of data-entry terminals.

    Jules L. DeVigne has served as Vice President of Worldwide Sales since
October 1992. He served as President of Innovative Technology, Inc., a
manufacturer of interactive voice response systems, from March 1990 until June
1992, and served as Senior Vice President of Sales and Marketing of Netrix
Corporation, a manufacturer of wide area networks, from February 1989 until
March 1990. Previously, Mr. DeVigne held various sales and executive management
positions with AT&T Paradyne Corporation, a manufacturer of modems and wide area
network products, and International Business Machines Corporation.

    Rubin Gruber was a founder of the Company and has served as a Director since
inception of the Company and as Vice President of Business Development since
February 1992. Mr. Gruber served as President of the Company from the Company's
inception until February 1992. He was a founder and served as President of Span
Communications, Inc., a development stage company, from August 1989 to August
1990. Previously, Mr. Gruber was a founder and served as President of both
Cambridge Telecommunications, Inc., a manufacturer of networking equipment, and
Davox Corporation, a developer of terminals supporting concurrent voice and data
applications, and as a Senior Vice President of Bolt, Beranek and Newman
Communications Corporation, a manufacturer of data communications equipment.

    Derek M. James was a founder of the Company, has served as Vice President of
Engineering since the Company's inception, and currently serves as the Chief
Technology Officer. He was Vice President of Engineering of Span Communications,
Inc., a development stage company, and an independent consultant from August
1989 until August 1990, and was Vice President of Product Development for Bolt,
Beranek and Newman Communications Corporation, a manufacturer of data
communications equipment, from August 1987 until August 1989. Previously, Mr.
James served as Vice President of Engineering of Decision Data, Inc., a
manufacturer of terminals and communications equipment, and Engineering Director
at Raytheon Data Systems, a manufacturer of data communications equipment.

    John Jones has served as Vice President of Customer Service since October
1995. He served as Director and General Manager of the Core Products Group at
Kronos, Inc., a manufacturer of time and attendance data collection systems from
April 1992 until October 1995, and served as General Manager of the Industrial
Applications Division of ModComp, an AEG Company, a manufacturer of
mini-computer systems and applications from June, 1989 until April 1992.
Previously, Mr. Jones held various technical and marketing positions with Apollo
computer, a manufacturer of technical workstations, and Data General
Corporation.

    Walter A. Jones has served as Vice President of Research and Development
since September, 1996. He served as Director of Engineering for the Isis
Distributed Systems Division at Stratus Computer, Inc., a manufacturer of fault
tolerant servers, from January 1994 until September 1996, and served as Vice
President of Engineering at Coral Networks, a manufacturer of high performance,
multi-protocol, network routers from April, 1992 until January 1994. Previously,
Mr. Jones served as Vice President of Engineering at Prime Computer, a
manufacturer of minicomputers and network servers.

    Stephen J. Nill has served as Vice President of Finance and Chief Financial
Officer since June 1994, and as Treasurer and Secretary since June 1995. He
served at Lotus Development Corporation, a software supplier, as Director of
Finance and Operations, Consulting Services Group, from October 1993 until May
1994, as Director of Worldwide Finance and Administration Systems from April
1993 until September 1993 and as Corporate


                                       12
<PAGE>   13
Controller and Chief Accounting Officer from January 1989 until March 1993.
Previously, Mr. Nill held various financial positions with Computervision, Inc.,
a supplier of workstation-based software, International Business Machines
Corporation and Arthur Andersen & Co.

    John E. O'Neil has served as Vice President of Marketing since February
1995. He served at Xylogics, Inc., a computer networking company, as Vice
President of Marketing from January 1992 until February 1995, and as Director of
Network Products Marketing from December 1988 until January 1992. Previously,
Mr. O'Neil held software engineering positions at Encore Computer, Inc., a
manufacturer of multiprocessor computer systems, and Datatrol, Inc., a supplier
of transaction-processing terminals.

     Frank T. Winiarski has served as Vice President of Manufacturing since June
1992. He served as Vice President of Manufacturing at Synernetics, a supplier of
local area networks, from July 1990 until May 1992, and as Executive Vice
President of The Lambda Group, a network hardware and software consulting firm,
from October 1989 until June 1990. Previously, Mr. Winiarski was Vice President
of Operations at Ashton-Tate Corporation, a software supplier, and held various
positions with Digital Equipment Corporation, a manufacturer of computer
equipment.

    Officers are elected on an annual basis to serve at the discretion of the
Board of Directors.

ITEM 2.  DESCRIPTION OF PROPERTY

    At December 31, 1996, the Company's principal offices were located in
Burlington, Massachusetts, in a 60,000 square foot facility which the Company
leases under agreements that expire in February 1999, with an option to renew
for two years.

    In March 1997, the Company relocated its European headquarters to a 4,500
square foot facility in Bracknell, United Kingdom, which the company leases
under a three year lease expiring in February, 2000. Also in March, 1997, the
Company entered into a lease for a 3,100 square foot facility in Dunwoodie,
Georgia, which will serve as a sales office, under a five year lease which
expires in February 2002.

    The Company also leases, on a short-term basis, various sales office space
in San Francisco, California; Reston, Virginia; and Dallas, Texas.

    The Company believes its existing and pending facilities are adequate for
its current needs and that suitable additional or substitute space will be
available as needed.

ITEM 3.  LEGAL PROCEEDINGS

    The Company is not currently involved in any legal proceedings that it
believes could have, either individually or in the aggregate, a material adverse
effect on its business or financial condition.

ITEM 4.  SUBMISSION OF MATTERS TO A VOTE OF SECURITY HOLDERS

    No matters were submitted to a vote of security holders during the last
quarter of the fiscal year ended December 31, 1996.


                                       13
<PAGE>   14
                                     PART II

ITEM 5.  MARKET FOR REGISTRANT'S COMMON EQUITY AND RELATED
         STOCKHOLDER MATTERS.

    The information required by this item may be found in the section captioned
"Quarterly Financial Information (unaudited)" appearing in the 1996 Annual
Report to Shareholders, and is incorporated herein by reference.(1)

    As of March 7, 1997, the Company had approximately 95 shareholders of
record. This does not reflect persons or entities who hold their stock in
nominee or "street" name through various brokerage firms. The Company has not
paid dividends on its Common Stock. The Company anticipates it will continue to
reinvest earnings to finance future growth, and therefore does not intend to pay
dividends in the foreseeable future.

ITEM 6.  SELECTED FINANCIAL DATA

    Information required by this item may be found in the section captioned
"Financial Highlights" appearing in the 1996 Annual Report to Shareholders, and
is incorporated herein by reference.(1)

ITEM 7. MANAGEMENT'S DISCUSSION AND ANALYSIS OF FINANCIAL CONDITION AND RESULTS
        OF OPERATIONS.

    Information required by this item may be found in the section captioned
"Management's Discussion and Analysis of Financial Condition and Results of
Operations" appearing in the 1996 Annual Report to Shareholders, and is
incorporated herein by reference.(1)

ITEM 8.  FINANCIAL STATEMENTS AND SUPPLEMENTARY DATA.

    Information with respect to this item may be found in the Financial Section
of the 1996 Annual Report to Shareholders on pages 22 through 32, and is
incorporated herein by reference.(1)

ITEM 9. CHANGES IN AND DISAGREEMENTS WITH ACCOUNTANTS ON ACCOUNTING AND
        FINANCIAL DISCLOSURE.

    None.

    ----------------------------------------------------------------------------
    (1) The Company's 1996 Annual Report to Shareholders is not to be deemed
filed as part of this report except for those parts thereof specifically
incorporated by reference.


                                       14
<PAGE>   15
                                    PART III

ITEM 10.  DIRECTORS AND EXECUTIVE OFFICERS OF THE REGISTRANT.

    Information with respect to Directors and compliance with Section 16(a) of
the Securities Exchange Act may be found in the sections captioned "Proposal No.
1 - Election of Director" and "Section 16(a) - Beneficial Ownership Reporting
Compliance" appearing in the definitive Proxy Statement to be delivered to
shareholders in connection with the Annual Meeting of Shareholders to be held on
May 14, 1997. Such information is incorporated herein by reference. Information
with respect to Executive Officers may be found under the section captioned
"Executive Officers of the Registrant" in Part I.

ITEM 11.  EXECUTIVE COMPENSATION.

    The information required with respect to this item may be found in the
sections captioned "Executive Compensation and Other Information Concerning
Directors and Executive Officers" appearing in the definitive Proxy Statement to
be delivered to shareholders in connection with the Annual Meeting of
Shareholders to be held on May 14, 1997. Such information is incorporated herein
by reference.

ITEM 12.  SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS AND MANAGEMENT.

    The information required with respect to this item may be found in the
section captioned "Security Ownership of Certain Beneficial Owners and
Management" appearing in the definitive Proxy Statement to be delivered to
shareholders in connection with the Annual Meeting of Shareholders to be held on
May 14, 1997. Such information is incorporated herein by reference.

ITEM 13.  CERTAIN RELATIONSHIPS AND RELATED TRANSACTIONS.

    The information required with respect to this item may be found in the
section captioned "Certain Transactions" appearing in the definitive Proxy
Statement to be delivered to shareholders in connection with the Annual Meeting
of Shareholders to be held on May 14, 1997. Such information is incorporated
herein by reference.



                                       15
<PAGE>   16
                                     PART IV

ITEM 14.  EXHIBITS, FINANCIAL STATEMENT SCHEDULES AND REPORTS ON FORM
               8-K.

(a) DOCUMENTS FILED AS PART OF FORM 10-K

1.       CONSOLIDATED FINANCIAL STATEMENTS.

         The following consolidated financial statements and supplementary data
         are included in Part II Item 8 filed as part of this report:

         -    Consolidated Balance Sheets as of December 31, 1995 and 1996


         -    Consolidated Statements of Income for the years ended December 31,
              1994, 1995 and 1996

         -    Consolidated Statements of Stockholders' Equity for the years
              ended December 31, 1994, 1995 and 1996

         -    Consolidated Statements of Cash Flows for the years ended December
              31, 1994, 1995 and 1996

         -    Notes to Consolidated Financial Statements

         -    Quarterly Financial Information (unaudited)

         -    Report of Independent Auditors

2.       FINANCIAL STATEMENT SCHEDULE.

         -    Schedule II - Valuation and Qualifying Accounts

         Schedules not listed above have been omitted because they are not
         applicable, not required or the information required is shown in the
         consolidated financial statements or the notes thereto.

3.       LIST OF EXHIBITS.

         Exhibit
         Number            Description of Exhibit

         3.1*              Form of Amended and Restated Certificate of
                           Incorporation of the Registrant.
         3.2*              Amended and Restated By-Laws of the Registrant.
         4.1*              Specimen Stock Certificate.
         10.1*             Amended and Restated 1991 Stock Incentive Plan of the
                           Registrant.
         10.2*             Amended and Restated 1994 Director Option Plan of the
                           Registrant.
         10.3*             1995 Employee Stock Purchase Plan of the Registrant.
         10.4*             Sublease for 5 Forbes Road, Second Floor, Lexington,
                           MA, dated as of May 1, 1991 between the Registrant
                           and Unitrode Corporation.
         10.5*             Lease for 5 Forbes Road, First Floor, Lexington, MA,
                           dated as of June 22, 1993 between the Registrant and
                           the Trustees of Lexington Development Company Trust.
         10.6*             First Amendment to Lease for 5 Forbes Road,
                           Lexington, MA, dated as of December 20, 1994 between
                           the Registrant and the Trustees of Lexington
                           Development Company Trust.
         10.7*             Unit Purchase Agreement, dated May 22, 1992 between
                           the Registrant and Edward Botwinick.


                                       16
<PAGE>   17
         10.8*             Series B Convertible Preferred Stock Agreement dated
                           as of August 10, 1992 and October 29, 1992 between
                           the Registrant and the purchasers named therein.

         10.9*             Series C Convertible Preferred Stock Agreement dated
                           as of March 28, 1994, May 16, 1994, and November 9,
                           1994 between the Registrant and the purchasers named
                           therein.

         10.10*            Noncompetition Agreement dated February 2, 1992
                           between the Registrant and Robert Castle.

         10.11*            Noncompetition Agreement dated March 28, 1991 between
                           the Registrant and Rubin Gruber.

         10.12*            Noncompetition Agreement dated March 28, 1991 between
                           the Registrant and Derek M. James.

         10.13*            Purchase and OEM License Agreement dated January 8,
                           1993 between the Registrant and Compression Labs,
                           Inc.

         10.14*            WorldWorx Personal Conferencing Service Multipoint
                           Control Unit Agreement dated February 15, 1995
                           between the Registrant and AT&T Corp.

         10.15*            License Agreement dated January 2, 1995 between the
                           Registrant and Datapoint Corporation.

         10.16*            Letter Agreement dated December 31, 1994 between the
                           Registrant and Fleet Bank of Massachusetts, N.A.


         10.17**           Lease for 63 Third Avenue, Burlington, MA dated as of
                           March 1, 1996 between the Registrant and the Trustees
                           of Building #27 Associates.

         11.1              Computation of Per Share Earnings.

         13.1              Financial Section of the 1996 Annual Report to
                           Shareholders, pages 17 through 32.


         21.1*             Subsidiaries of the Registrant.

         23.1              Consent of Ernst & Young LLP.

                           (Note: The Company agrees to furnish to the
                           Securities and Exchange Commission upon request a
                           copy of any instrument with respect to long-term debt
                           of the Company or any of its subsidiaries which is
                           not filed herewith or listed herein since it relates
                           to outstanding debt in an amount not greater than 10%
                           of the total assets of the Company and its
                           subsidiaries on a consolidated basis.)

         *   Incorporated by reference from the Company's Registration Statement
             on Form S-1.
         **  Included as part of the Company's Form 10-K filed with the
             Securities and Exchange Commission for the year ended December 31,
             1995.

(b) REPORTS ON FORM 8-K

         The Company filed no reports on Form 8-K during the quarter ended
December 31, 1996.


                                       17
<PAGE>   18
                                   SIGNATURES


Pursuant to the requirements of Section 13 or 15(d) of the Securities Exchange
Act of 1934, the registrant has duly caused this report to be signed on its
behalf by the undersigned, thereunto duly authorized.

                                          VIDEOSERVER, INC.


                                          /s/ Robert L. Castle
                                          -------------------------------------
                                          Robert L. Castle
                                          President and Chief Executive Officer
                                          (Principal Executive Officer)
                                          Date: March 28, 1997

Pursuant to the requirements of the Securities Exchange Act of 1934, this report
has been signed below by the following persons on its behalf of the registrant
and in the capacities and on the dates indicated.

<TABLE>
<CAPTION>
SIGNATURE                                      TITLE                    DATE
---------                                      -----                    ----
<S>                               <C>                                   <C>
/s/ Robert L. Castle              President, Chief Executive Officer,   March 28, 1997
---------------------             (Principal Executive Officer)
Robert L. Castle                  and Director


/s/ Rubin Gruber                  Vice President of Business             March 28, 1997
---------------------             Development and Director
Rubin Gruber


/s/ Stephen J. Nill               Vice President and                     March 28, 1997
---------------------             Chief Financial Officer
Stephen J. Nill                   (Principal Financial
                                  and Accounting Officer)


/s/ Paul J. Ferri                 Director                               March 28, 1997
---------------------
Paul J. Ferri


/s/ William E. Foster             Director                               March 28, 1997
---------------------
William E. Foster


/s/ Steven C. Walske              Director                               March 28, 1997
---------------------
Steven C. Walske
</TABLE>

                                       18
<PAGE>   19
VIDEOSERVER, INC.
SCHEDULE II - VALUATION AND QUALIFYING ACCOUNTS

<TABLE>
<CAPTION>
Column A - Description                 Column B         Column C - Additions               Column D          Column E
----------------------                 --------         --------------------               ---------         --------
                                                                                           Deductions -
                                       Balance At       Charged to        Charged to       Uncollectable     Balance at
                                       Beginning Of     Costs and         Other            Accounts          End of Period
Allowance for Doubtful Accounts        Period           Expenses          Accounts         Written Off
------------------------------------   --------------   ---------------   ---------------  ---------------   ---------------
<S>                                    <C>              <C>               <C>              <C>               <C>
  Year Ended December 31, 1996              $650,313          $427,140          $-               $-              $1,077,453

  Year Ended December 31, 1995               118,509           531,804           -                -                 650,313

  Year Ended December 31, 1994                 6,000           112,509           -                -                 118,509
</TABLE>


                                       19