METHOD AND APPARATUS FOR PROVIDING SECURITY IN A STAR NETWORK CONNECTION USING PUBLIC KEY CRYPTOGRAPHY

Abstract: An intermediate system authenticates end systems, in one embodiment using cryptography. An authentication routine is called on a port when an intermediate system detects any interruption in the physical connection with the port. An embodiment requires a user to supply a secret known only to the user before allowing data to be transmitted. Network data is not fully transmitted or received to any port that has not been authenticated. The invention distributes a user authentication to the point where an end system initially connects to a network, to prevent an authorized reception or transmission of network data that is not prevented under existing network login systems. The invention may be incorporated into an intermediate system, into intermediate system software, or into applications specific integrated circuits designed for use in an intermediate system. The invention may include components that interact specifically with installed components in an end system or elsewhere in a network.

Description: METHOD AND APPARATUS FOR PROVIDING SECURITY IN A STAR NETWORK CONNECTION USING PUBLIC KEY CRYPTOGRAPHY BACKGROUND OF THE INVENTION This application claims priority from U.S. patent application 08/866,818 entitled METHOD AND APPARATUS FOR PROVIDING SECURITY IN A STAR OR HUB NETWORK CONNECTION filed in the U.S. May 30, 1997 (9704-92) and from its continuation in part U.S. patent application 08/955,869 entitled METHOD AND APPARATUS FOR PROVIDING SECURITY IN A STAR NETWORK CONNECTION USING PUBLIC KEY CRYPTOGRAPHY filed in the U.S. October 28, 1997 (9?64-9 6 ). Related disclosure is discussed in co-assigned patent application serial number 08/733,358, filed 10/17/96, which is a continuation of U.S. Patent 5,568,469, filed 9/27/94 and issued 10/22/96. The current invention relates to the field of electronic circuits. More particularly, the current invention relates to improvements in networked computer environments and has particular applications to the transmission of information between digital devices over a communications medium. A wide variety of computer systems and networks exist, each having variations in particular implementations. The present invention will be described with reference to particular types of systems for clarity but this should not be taken to limit the invention, and it will be apparent to those of skill in the art that the invention has applications in many different types of computer systems. The invention therefore should not be seen as limited except as specifically herein provided. Networking Devices Standards This specification presumes familiarity with the general concepts, protocols, and devices currently used in LAN networking and WAN internetworking applications such as, for example, the IEEE 802 and ISO 8802 protocol suites and other series of documents released by the Internet Engineering Task Force that are publicly available and discussed in more detail in the above-referenced patent applications and will not be fully discussed here. ____! Fig. 1 illustrates a local area network (LAN) 40 of a type that might be used today in a moderate-sized office or academic environment and as an example for discussion purposes of a system in which the present invention may be effectively employed. LANs are arrangements of various hardware and software elements that operate together to allow a number of digital devices to exchange data within the LAN and also may include internet connections to external wide area networks (WANs) such as WANs 82 and 84. Typical modern LANs such as 40 are comprised of one to many LAN intermediate systems (ISs) such as ISs 60-62 and 67 that are responsible for data transmission throughout the LAN and a number of end systems (ESs) such as ESs 50a-e, 51a-c, and 52a- , that can represent end user equipment or other end nodes. Fig. 1 also shows a security server 50f and a directory server 52g which, according to embodiments of the invention as described below, are used by a star device to authenticate new connections. The ESs may be familiar end-user data processing equipment such as personal computers, workstations, and printers and additionally may be digital devices such as digital telephones or real-time video displays. Different types of ESs can operate together on the same LAN. In one type of LAN, LAN ISs 60-61 are referred to as bridges and WAN ISs 64 and 66 are referred to as routers, and IS 67 is referred to as a repeater, however many different LAN configurations are possible, and the invention is not limited in application to the network shown in Fig. 1. The LAN shown in Fig. 1 has segments 70a-g, 71a-e, and 72a-e, and 73a. A segment is generally a single interconnected medium, such as a length of contiguous wire, optical fiber, or coaxial cable or a particular frequency band. A segment may connect just two devices, such as segment 70a (also referred to as a point-to-point connection or a star connection), or a segment such as 72d may connect a number of devices using a carrier sense multiple access/collision detect (CSMA/CD) protocol or other multiple access protocol such as a token bus or token ring. A signal transmitted on a single segment, such as 72d, is simultaneously heard by all of the ESs and ISs connected to that segment. LANs also may contain a number of repeaters, such as repeater 67. A repeater generally physically repeats out of each of its ports all data received on any one port, such that the network behavior perceived by ESs 50a-c and the port of IS 60 connected to 67 is identical to the behavior these ports would perceive if they were wired on the same segment such as 52d-g and the corresponding port of 62. Repeaters configured in a star topology, such as 67, are also referred to as hub repeaters. (The terms hub or star is a term used in networking that sometimes indicates either a switch/bridge layer 2 device or a repeater layer 1 device depending on the context.) In the network shown in Fig. 1, bridges 61, 62, and 63 can be considered to have a star or hub configuration, as can repeater 67. The term node is often used in describing networks to indicate any transmitter or receiver in the network. Repeater 67, for example, is a network intermediate system connected to four nodes, three of those being end systems and one being a different intermediate system bridge 60. Drivers, Adaptors, and LAN Topology Each of the ISs and ESs in Fig. 1 includes one or more adaptors and hardware or software instructions sometimes referred to as drivers. An adaptor generally includes circuitry and connectors for communication over a segment and translates data from the digital form used by the computer circuitry in the IS or ES into a form that may be transmitted over the segment, e.g., electrical signals, optical signals, radio waves, etc. An ES such as 50b will generally have one adaptor for connecting to its single segment. A LAN IS such as 61 will have five adaptors, one for each segment to which it is connected. A driver is a set of instructions resident on a device that allows the device to accomplish various tasks as defined by different network protocols. Drivers are generally software programs stored on the ISs or ESs in a manner that allows the drivers to be modified without modifying the IS or ES hardware. LANs may vary in the topology oϊ the interconnections among devices. In the context of a communication network, the term "topology" refers to the way in which the stations attached to the network are interconnected. Common topologies for LANs are bus, tree, ring, and star. LANs may also have a hybrid topology made up of a mixture of these. The overall LAN pictured in Fig. 1 has essentially a tree topology, but incorporating one segment, 72d, having a bus topology, and incorporating one segment 70d having a star topology. A ring topology is not shown in Fig. 1, but it will be understood that the present invention may be used in conjunction with LANs having a ring topology. Network Intermediate Systems: Routers, Bridges, Repeaters The LAN ISs in LAN 40 include bridges 60-63. Bridges are understood in the art to be a type of computer optimized for very fast data communication between two or more segments. A bridge according to the prior art generally makes no changes to the packets it receives on one segment before transmitting them on another segment. Bridges are not necessary for operation of a LAN and, in fact, in prior art systems bridges are generally invisible to the ESs to which they are connected and sometimes to other bridges and routers. LANs also may contain a number of repeaters, which is one configuration possible for device 67. A repeater generally repeats out of each of its ports all data received on any one port, such that the network behavior perceived by ESs such as 50d-f is generally identical to the behavior they would perceive if they were wired on the same segment such as 52d-g. In alternative network topologies, device 67 could be a bridge/switch as described in related patent applications. Layers An additional background concept important to understanding network communications is the concept of layered network protocols. Modern communication standards, such as the TCP/IP Suite and the IEEE 802 standards, organize the tasks necessary for data communication into layers. At different layers, data is viewed and organized differently, different protocols are followed, and different physical devices handle the data traffic as abstracted at that layer. Fig. 3 illustrates one example of a layered network standard having a number of layers, which we will refer to herein as the Physical Layer, the Data Link Layer, the Routing Layer, the Transport Layer and the Application Layer. These layers correspond roughly to the layers as defined within the TCP/IP Suite. (The 802 standard has a different organizational structure for the layers and uses somewhat different names and numbering conventions.) An important ideal in layered standards is the ideal of layer independence. A layered protocol suite specifies standard interfaces between layers such that, in theory, a device and protocol operating at one layer can coexist with any number of different protocols operating at higher or lower layers, so long as the standard interfaces between layers are followed. Network modularity creates increasing security concerns In many office environments, networks such as that shown in Fig. 1 are designed to be highly modular. When a office building is constructed, typically network jack connections are provided throughout the office space, allowing computers to be flexibly installed in various offices or moved around as needed. In such a case, any of the ESs shown in Fig. 1 may represent unconnected jacks awaiting connection to an actual ES computer. While this modularity is highly desirable in many office environments, it also creates security concerns in that a person with access to a workspace containing a network jack can connect a device to the jack and that device can hear some or all of the traffic on the network. Furthermore, that device can potentially even send traffic out over the network to be routed via a gateway over the internet. In a worse-case scenario, a small box could be plugged into a network jack and forward all LAN traffic over the network's WAN connection to a remote location without a network owner being aware of the breach in security. Another potential security problem in a network can occur when an authorized network end system, that is properly installed, is not attended for a period of time and can be accessed by an unauthorized person. In such a case, even where a user is unable to login to the network server, an unauthorized person could run software on the end system that would be capable of sending out packets on the network or of sniffing packets on the network, thereby gaining unauthorized access to network data or resources. In prior art network systems, unauthorized access to a network server is prevented by requiring users to login to the server and by timing out an idle login either at the server or setting up a password-protected Screensaver. However, neither of these mechanisms prevents an unauthorized person from rebooting an ES with a network connection or powering up a turned-off ES and thus gaining access to the network. Some prior art systems also include user verification at the point of a telephone dial-in SLIP or PPP connection to a network, but these systems are limited to a dial-in environment. What is needed is a method and apparatus for improving security in a LAN and for preventing unauthorized equipment or an unauthorized person from connecting to a network and listening-in on network traffic. What is also needed is a method and apparatus for verifying the identity of a user prior to allowing the user to send or receive data on a network. The parent U.S. application discusses mechanisms of providing increased network security using link-beat detection at a star intermediate system and a variety of authentication schemes that are called whenever an unauthenticated user connects or reconnects to the star device. One aspect of that invention involves a star system requiring a user to supply some type of password to the star system for authentication. However, in some applications it will be desirable to provide a system wherein a user's password is not transmitted over the network so that the password cannot be captured by a sniffing device. What is further needed, therefore, is a user authentication system that does not require that a user password be transmitted on the network but prevents unauthorized equipment or an unauthorized person from connecting to a network and listening-in on network traffic and verifies the identity of a user prior to allowing the user to send or receive data on a network. SUMMARY OF THE INVENTION In general terms, the present invention comprises techniques and devices for a computer network with improved security. According to the invention, network intermediate systems (IS) that are connected in a star or similar topology, such as repeaters or switches, that provide point-to-point connections to one or more ESs are enabled to authenticate a user at the first connection point the user makes with the network. The intermediate systems provide an authentication routine every time an ES makes a new connection on a port. According to the invention, an IS will limit the forwarding of packets down its port to an ES and will limit forwarding data from an ES until an authentication process is complete. A new connection to a point-to-point port on an IS may be detected by the presence or resumption of a link-beat (sometimes referred to as heart-beat) signal that is maintained at the physical layer on the point-to-point connection whenever there is an active node/ES on a port. Authentication is accomplished at a user level through a challenge/response exchange as described below. Authentication according to the invention is controlled by a first network intermediate system (star device) to which an end system connects. Some parts of the authentication may be performed locally and, in an alternative embodiment, the star device may communicate with one or more other devices on the network to complete authentication of a new user. According to a further embodiment of the invention, at power up or when a star device detects a new connection on a port, the star device initiates an authentication routine to reliably verify a user. To do this, a star device, according to one embodiment, may use a directory server existing somewhere on the network to retrieve a certificate or a certification path for the user based on an identification supplied by a user. The certificate or certification path is decoded into a public key for a particular user. The invention uses a modified public key cryptography and a challenge response scheme to authenticate a user. In one embodiment, once a public key for a particular user has been decoded, a challenge is generated and is encrypted with the user's public key. The encrypted challenge is then sent to the end system. The user must then supply his private key to decrypt the challenge at the end system and the end system returns the decrypted challenge to the star device. The invention compares the response with the challenge it originally generated and based on the results of the match, authenticates the port. In an alternative embodiment, a challenge is generated and presented to the user, and the user then encrypts the challenge using his private key and returns the encrypted challenge as a response. A public key for a particular user it then used to decrypt the challenge, either at the star device or the security server. The invention compares the decrypted response with the challenge it originally generated and based on the results of the match, authenticates the port. In a further embodiment, if the star device is restricted in terms of the computation power it can devote to authentication, a security server performs the computation-intensive cryptography functions. The star device in this case receives the user name from the end system and requests the security server to compute the challenge for the user. The security server gets the user certification path, generates a challenge, encrypts it using the user's public key and sends both the challenge and encrypted challenge to the star device. The star device then sends only the encrypted challenge to the end system for decryption. When the end system responds with the decrypted challenge, the star device compares the returned challenge with the unencrypted original challenge supplied by the security server and allows or disallows network access to the user based on the comparison results. In this embodiment, it is important that the connection between the star device and security server be secure such that an intruder cannot capture the challenge and encrypted challenge. This may be done by having the security server directly coupled or integrated with the star device. Alternatively, messages using message integrity and confidentiality mechanisms, as are known in the art, can be used to ensure that the message from the security server to the star device is generated by the security server. According to a further embodiment the invention may be incorporated in a network intermediate system with a star configuration which may perform authentication locally or in a preferred embodiment may communicate with a login server on the network to authenticate verification data which may consist of a user name and password of a user or of a challenge response data or other verification data supplied by the user. Specific aspects of the invention will be better understood upon reference to the following detailed description and in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram of a moderate sized network of one type in which the invention may be effectively employed; FIG. 2 is a diagram illustrating an IS connecting four transmitter/receivers in a star/hub connection to illustrate aspects of the invention; FIG. 3 is a diagram illustrating a layered network protocol; FIG. 4 is a block diagram of an IS star device according to one specific embodiment of the invention; FIG. 5 is a flow chart illustrating user authentication according to one embodiment of the invention; FIG. 6 is a diagram of a computer system with a fixed medium 717 which may be used to implement one embodiment of the invention. DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS Fig. 2 is a generalized and simplified block diagram of a star device IS connected to three ESs and one other IS that will be used to illustrate the invention. The star IS 67 may represent either a repeater at layer 1 or a bridge at layer 2 or other network intermediate systems according to different network configurations. Fig. 4 is a block diagram representing an IS internal structure according to one embodiment of the invention. As discussed above, in one embodiment, a star device according to the invention maintains state regarding which of its connected ports are authenticated and which are not and detects when the node connected to a port is disconnected. An authentication process is called whenever a physical connection is detected on an unauthenticated port or whenever any attempt is made to transmit data on an unauthenticated port. In one embodiment, network transmission is limited on a port until a port connection has been authenticated. Fig. 4 also shows security server 95. According to one embodiment of the invention, security server 95 is supplied with a user identification from IS 67 and then presents to IS 67 an encrypted challenge and an unencrypted challenge for performing authentication as described below. Security server 95 may be a process running on the same hardware as IS 67 or may be a separate device on the network such as server 50f shown in Fig. 1. Security server 95 performs a type of two-key encryption in engine 97 using a user's "public" key, as is known in the art, after first retrieving a user's public key using retrieval engine 99. Public key encryption uses a particular receiver's public key to encrypt a message. That encrypted message can then be decrypted only by a matching private key, which is generally known only to the owner of the private key. In one embodiment of the invention, the authenticity of a retrieved public key is guaranteed by using a hierarchical certification path and certification authority as is known in the art. The certificate path may be stored along with other user configuration data in a directory server 52g, which may then act as one of the certification authorities on the network. Physical Layer Linkbeat Detection In contrast to prior art network and LAN systems, an embodiment of the present invention authenticates a user at a first LAN intermediate system to which the user is connected on a per port basis, then authenticates a user each time there is any interruption in the physical link with an end system connected to a particular network port. In order to accomplish this, the invention uses a mechanism at the physical layer to determine whether an end system is newly connected to a port, rebooted, or power cycled at a port. In one specific embodiment, point-to-point ethernet linkbeat, as is known in the art, is used to determine whether there has been any interruption in the physical connection to an end system. As is known in the art, point-to-point ethernet defines a linkbeat packet whereby, during periods of silence, adapters at both ends of a point-to- point ethernet connection send periodic very small packets, called linkbeat packets, to inform the adapter at the other end that an adapter is connected to a port. In prior art systems, the presence or absence of this linkbeat packet is used only at the lowest physical layer to manage transmissions and is not made known to higher layer protocols. According to the current invention, however, a controller, such as 82, is enabled to respond to an interruption in the linkbeat on a port by resetting an authentication bit in a port table 85. A resumption of linkbeat on a port or an attempt to send data on a port that is unauthenticated will be detected by controller 82 and will cause the controller to initiate an authentication process 89 as described below. Other mechanisms for detecting the interruption in a physical connection are possible for use in conjunction with the invention, and according to the invention, any mechanism appropriate for a particular physical interface that can sense a disconnection from the physical interface or a reboot of an end system connected to the physical interface may be used with the authentication scheme of the invention. Authenticating a Port According to the invention, an authentication process is called generally whenever data transmission is attempted or linkbeat is resumed on an unauthenticated port. According to an embodiment, when authentication process 89 is invoked by controller 82, the authentication process will cause a message to be sent to an unauthenticated end system, such as 50a requesting authentication data. Such data may be in the form of a user name and password, as is known in the art, or in the form of a challenge/response protocol, where the intermediate system selects a challenge value and a user must provide a corresponding value according to a secret code. In response, the end system 50a transmits authentication data back to the intermediate system 67 for authentication. Intermediate system 67 then uses that authentication data to authenticate the user on an unauthenticated port. This may be done locally, in which case the intermediate system 67 will keep an authentication table local to it or it may be done using a remote login server as is known in the art. In the latter case, authentication process 89 will compose an appropriate protocol message to the login server, and that message will be transmitted over the local area network. As is known in the art, the login server may be any processor running on the network that can receive authentication messages. The login server then transmits back a message indicating whether the authentication data is valid when compared against data stored at the login server. This message is received by authentication process 89. In response thereto, authentication process 89 can set the authentication bit in port table 85, indicating that the port is authenticated. In one specific embodiment, the authentication process takes place according to the flow chart shown in Fig. 5. According to this embodiment, when authentication process 89 is invoked by controller 82, the authentication process will cause a message to be sent to an unauthenticated end system, such as 50a requesting an identification that will be used for authentication (referred to herein as a user id). The user id is then provided to a security server 95, which is a process accessible to device 67, either running on that device or elsewhere on the network. In one embodiment, it is the task of security server 95 to generate both a challenge to be presented to an end system and an expected response from the end system. In an alternative embodiment, the star device will generate the challenge and the security server will generate an expected response. Separating out the task of generating a challenge/expected response to a stand-alone security server has the advantage of moving this computationally demanding task off of the star device. In order for the security server to encrypt the challenge, it must have a public key for the user. In one embodiment, the security server looks up a public key for a user using a directory server or other data repository, which is generally a remote server on the network that stores a variety of data about a user. The security server may also store the public key locally and may also cache public keys for known users for a period of time so that repeated logins by a user at an end system connected to the same security server do not require full fetching and decoding of a certification path from a directory server. In one embodiment, the authentication process will require a user to have its private key available to decrypt a new challenge each time the authentication process is run rather than allowing the process to be run using a stored password at an end system. system. In this way, access to the network can be prevented by an authorized person using an authorized and properly connected end system. FIG. 5 depicts a flow chart of an authentication method according to the invention. The method begins when an intermediate system detects a new user or connection of a port (SI). The IS requests a user identification from the ES (S2) and provides that identification to a security server (S3). The security server retrieves a certification path from a certification authority (S4) and derives from that a public key (Pk), which it then uses to encrypt a challenge to produce an expected response (Pk(Rn)) (S5). The security server supplies a challenge and an encrypted challenge to a star device (S6), which presents the encrypted challenge to the ES (S7). The ES must then use its private key to decrypt the encrypted challenge and return a response to the star device (S8), which then compares the response to the unencrypted challenge to determine if it should authenticate the port (S9). In an alternative embodiment of the invention, an unencrypted challenge is presented to the end-system, which must then use its private key to encrypt the challenge, which is returned to the star device. Then either the star device, or a security server, uses a public key for the user to decrypt the response and compares that to the original challenge. Controlling Access to the Network An intermediate system, such as 67, with a direct connection to a number of end systems prevents unauthorized access to a network by controlling the flow of network data to end systems that have not been authenticated. An intermediate system, such as 67, may simply allow no data to flow to and from an end system that is unauthenticated, except for the minimum data needed to interact with the authentication process 89, requiring that an end system be configured such that the authentication may be accomplished prior to any other access to a network being necessary. An unauthenticated end system may also be allowed limited access to the network in order to accomplish initial network login and configuration prior to completing the authentication process. One mechanism for doing so is to include a protected field in a port table, such as 85, allowing an unauthenticated port to transmit data out of an unprotected port only and remaining restricted until an authentication process is complete. Intermediate system 67 may also include a further filtering process for unauthenticated ports, allowing packets to flow only to particular addresses or restricting the protocol of packets flowing to or from any unauthenticated port. In such case, a filtering process, such as 90, may be invoked only on unauthenticated ports to examine either addresses or protocols of packets to and from such ports and to prevent the flow of packets that do not meet the filter process definitions. A filtering process may also be invoked on packets in accordance with a Virtual LAN architecture, such as described in co-assigned patent applications serial number 08/502,835, entitled VIRTUAL NETWORKING ARCHITECTURE filed July 14, 1995, and serial number 08/ , , entitled VIRTUAL NETWORKING ARCHITECTURE FOR CONNECTIONLESS LAN BACKBONE. In a further embodiment, the invention is designed to work in a managed intermediate system, as is known in the art, managed by a standard network protocol such as SNMP or other network protocol. In such a system, a network manager may set various authenticating characteristics. In a further embodiment, the invention may network with an intermediate system incorporating an end-system detecting algorithm, allowing the intermediate system to detect whether a single end system is connected to a port, such as port A, or another intermediate system, such as a repeater, is connected to a port. In a further embodiment, the invention is designed to require modifying an intermediate system only. In such a case, an intermediate system according to the invention works with preexisting adapter driver software and operating system software on end systems to call the authentication routines on an end system as defined by the preexisting operating system at the appropriate time. In other embodiments, the invention will work with a specially designed adapter driver software on an end system, to perform the authentication routine. The invention may be embodied in a set of executable computer program code which may be stored into a fixed computer medium such as a disk, diskette, volatile memory or nonvolatile memory, or any other medium for storing computer code. In such a case, when such instructions are loaded and executed in an appropriately configured network intermediate system, the intermediate system will perform as described herein. A representation of such a system 700 is shown in Fig. 5 containing CPU 707, optional input devices 709 and 711, disk drives 715 and optional monitor 705. Fixed media 717 may be used to program such a system and could represent a disk-type optical or magnetic media or a memory. The invention also may be embodied within the circuitry of an application specific integrated circuit (ASIC). In such a case, the invention may be embodied in a computer understandable descriptor language which may be used by an ASIC foundry to create an ASIC device that operates as herein described. In this embodiment, computer system 700 may be understood as a computer system for reading instructions from media 717 for the manufacturing of an ASIC. The invention has now been explained with reference to specific embodiments. Other embodiments will be obvious to those with skill in the art. In particular the invention has application in many different networking environments and schemes. It is therefore intended that the invention not be limited except as indicated in the attached claims.

Claims: WHAT IS CLAIMED IS: 1. In a network, a method for an intermediate system limiting transmission or reception of data with an unauthorized user comprising: in a network intermediate system having a plurality of ports for connection to network segments, storing an indication for each port indicating whether said port is authenticated or unauthenticated; detecting when an end-system is not present or is rebooted on a port and setting that port to an unauthenticated state; prior to freely transmitting or receiving network data on said port, authenticating said port. 2. The method according to claim 1 wherein said step of authenticating a port comprises receiving user verification data from said port and verifying said verification data and wherein said step of detecting comprises detecting the resumption of link-beat on a port. 3. The method according to claim 2 wherein said step of verifying comprises transmitting verification data to a login server on said network and receiving a confirmation from said login server. 4. The method according to claim 2 wherein said step of verifying comprises presenting a challenge on said port requesting that said port provide a secret value that would only be known to an authorized user. 5. The method according to claim 2 wherein after verification on said port verification data is used by an operating system on said end-system to verify a user to other network resources. 6. The method according to claim 1 wherein said intermediate system allows a limited amount of data to be transmitted and received on a port prior to authentication of that port. 7. The method according to claim 1 further comprising limiting data transmission with an unauthenticated port to a subset of other ports that are designated unprotected ports. 8. A method for improving network security comprising: at a first intermediate system to which a network end system connects storing state indicating whether a network port connecting said intermediate system to an end system is authenticated; initially setting said state to an unauthenticated state at intermediate system power up; prior to an unrestricted transmission or reception of network data through said intermediate system of an unauthenticated port, performing an authentication process on said port; resetting said authentication state to an unauthenticated state whenever an interruption in the physical connection between a port and an end system is detected; and setting said state to an authenticated state only when said authentication process returns a message that authentication has been validated. 9. A method for authenticating a user by a network comprising: requesting an identification; using said identification to determine a public encryption key for said user; generating a challenge by said network for said user; presenting said challenge to said user; receiving from said user a response to said challenge, said response generated at said user by operating on said challenge using a private key; verifying said response by performing an operation on said response and said challenge; and authenticating said user only if said verifying indicates that said private key corresponds to said public key. 10. The method according to claim 9 wherein said step of requesting comprises requesting an identification to be supplied by a user said identification possibly shared with or passed from other login routines. 11. The method according to'claim 9 wherein said step of requesting comprises prompting a user for a user name. 12. The method according to claim 9 wherein said step of using said identification comprises passing said identification to a security server to determine a public key for said user. 13. The method according to claim 9 wherein said step of using said identification comprises using said identification to look up a certificate and decoding that certificate to determine a public key for said user. 14. The method according to claim 9 wherein said step of using said identification comprises transmitting an identification to a directory server said directory server retrieving a certification path that corresponds to said identification and returning said certification path. 15. The method according to claim 9 further comprising: computing a response at an end system from said challenge, said response requiring a user to supply a private key at an end system wherein said step of computing comprises using a private key for said user and public-key/private-key encryption algorithm to generate said response. 16. The method according to claim 9 wherein said step of generating a challenge comprises generating a random number. 17. The method according to claim 9 wherein said step of generating a challenge comprises using said public key and public-key/private-key encryption algorithm to generate said challenge by encrypting a message, said challenge comprising said encrypted message. 18. The method according to claim 9 wherein said step of generating a challenge is performed by a security server. 19. The method according to claim 9 wherein said step of verifying comprises decrypting said response using said public key and comparing the result to said challenge. 20. The method according to claim 19 wherein said decrypting of said response is performed by a security server. 21. A fixed computer-readable medium containing computer program code that when loaded into an appropriately configured network intermediate system and run will cause the computer to perform any of the method claims 1 through 20. 22. A fixed computer-readable medium containing computer- interpretable instructions describing a circuit layout for an integrated circuit that, when constructed according to said descriptions and placed into an appropriately configured computer system, will cause the computer to perform any of the method claims 1 through 20. 23. A secure hub comprising: a port state variable containing state for a port indicating whether an end system connected to said port has been authenticated; an authentication process capable of receiving a user identification from a port and of communicating with network devices in order to receive a public key for said user and challenge data to present to a port; a verification engine for determining if a response is valid by operating on said response and said challenge using said public key and a public key/private key encryption algorithm; and a controller capable of setting an authentication bit for a port in response to a result from said comparator. 24. The device according to claim 23 further comprising a second state indicating whether at least one port is an unprotected port out of which data from unauthenticated ports may be transmitted. 25. A network having enhanced security comprising: an end system interacting with a user; a state variable containing state indicating whether a connected end system has been authenticated; a secure hub providing a private connection between said end system and network data and capable of filtering data to said end system based on an authentication state for said end system; an authentication process capable of receiving a user identification from an end system and capable of retrieving a mechanism for generating a challenge to present to an end system; a comparator for comparing a response from an end system to a challenge, the validity of said response depending on a secret key being known at said end system; and a controller capable of setting an authentication bit for an end system in response to a result from said comparator.

Inventors: JAIN VIPIN KUMAR, NESSETT DANNY M, SHERER WILLIAM PAUL, JAIN, VIPIN, KUMAR, NESSETT, DANNY, M., SHERER, WILLIAM, PAUL
Applicants: 3COM CORP, 3COM CORPORATION
Application year: 1998
Main classification: H04L12
IPC codes: H04L9, H04L12, H04L29