Method and apparatus for operating a network mapping tool to perform host discovery

In accordance with one aspect of the embodiments of this invention there is a method to operate a network mapping tool. The method includes performing a network mapping operation over a first set of ports to generate a list of hosts that are found to be alive; performing a sequential scan on only those hosts in the list to detect the operating system of each host in the list; and performing a parallel port scan to detect characteristics of only those hosts in the list. The characteristics can include an identification of at least applications and services running on the hosts in the list. The second set of ports can be a full set of ports.

TECHNICAL FIELD

The embodiments of this invention relate generally to data processing systems and networks and, more specifically, relate to techniques for the automated discovery and characterization of communication network resources.

BACKGROUND

The following abbreviations that appear in the description and/or the drawing figures are defined as follows:

FW firewall

HTTP hypertext transfer protocol

IT information technology

LANA internet assigned numbers authority

IP internet protocol

NMAP network mapper

OS operating system

RMIS remote managed infrastructure services

TCP transport control protocol

UDP user datagram protocol

A “port” may be considered as a logical connection point for a computer system or program. When considered in the context of TCP/IP a port can be considered to represent the doorway by which a client program may interact with or exchange information with a particular server program on a computer in a network. In this case the client program is said to use the server program's port number N. Certain applications that use TCP/IP, such as HTTP, have ports with pre-assigned numbers. The entire computing community accepts these ports as a standard and agrees not to use them for private purposes. These are generally referred to as “well-known ports”, and have been pre-assigned by the IANA. Other application processes can be dynamically assigned port numbers for each connection. When a service initially is started, it is said to bind to its designated port number. Reference with regard to port numbers can be made to, for example, RFC 768 and RFC 793

The port numbers are divided into three ranges: the well known ports, registered ports, and dynamic and/or private ports. The well known ports are those from 0 through 1023, the registered ports are those from 1024 through 49151, and the dynamic and/or private ports are those from 49152 through 65535. For HTTP service port 80 is defined as a default port number.

In IP specifications the term “host” generally implies a computer that has bi-directional access to other computers on the Internet. A host has a specific host number that, together with the network number, forms its unique IP address. In some contexts a host may be considered to be a node on a network.

Network mapping is used to determine computers or servers, or more generally hosts that are running on a network, and the programs running on the hosts. There are a number of different network mapping tools and programs that are in use.

On example of a network mapping tool is NMAP. NMAP is an open source tool for network exploration and security auditing. NMAP is capable of scanning large networks as well as single hosts. NMAP uses raw IP packets to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what OSs (and OS versions) the hosts are running, what type of packet filters/FWs are in use, and many other network characteristics. While NMAP is commonly used for security audits it is also useful for other tasks such as network inventory, managing service upgrade schedules and monitoring host or service uptime.

By default, NMAP scans items listed in a command line as IP addresses or range of IP addresses on a predefined range of port numbers and attempts to identify the OS, applications (if given a “-sV” option) and services on the active ports.

NMAP OS fingerprinting operates by sending up to 16 TCP, UDP, and ICMP probes to known open and closed ports of the target machine. These probes are specially designed to exploit various ambiguities in the standard protocol RFCs. NMAP then listens for responses to these probes. Dozens of attributes in the responses are analyzed and combined to generate a fingerprint. Every probe packet is tracked and re-sent at least once if there is no response. All of the packets are IPv4 with a random IP ID value. Probes to an open TCP port are skipped if no such port has been found. For closed TCP or UDP ports, NMAP first checks if such a port has been found. If not, NMAP will select a port at random.

However, the approach taken by the conventional use of NMAP proves to be error prone in cases when parallel scanning is involved, which is important for response time performance when the NMAP results are used interactivity. Successive parallel scans on the same host can lead to the “discovery” of different OSs, since NMAP is not always able to take a clear fingerprint of the OS. Referring to, for example, an on-line NMAP book: Chapter 8: Remote OS Detection: Dealing with Misidentified and Unidentified Hosts, one suggestion made for improving results is to scan all ports (using a -p-option in the command line).

As can be appreciated, a need exists to improve the accuracy and repeatability of network scanning tools.

SUMMARY

The foregoing and other problems are overcome, and other advantages are realized, in accordance with the exemplary embodiments of this invention.

In one aspect of the embodiments of this invention there is a method to operate a network mapping tool, comprising performing a network mapping operation over a first set of ports to generate a list of hosts that are found to be alive; performing a sequential scan on only those hosts in the list to detect the operating system of each host in the list; and performing a parallel port scan to detect characteristics of only those hosts in the list.

In another aspect of the embodiments of this invention there is a computer-readable storage medium having computer-executable program code stored thereon or therein. Execution of the program code results in operating a network mapping tool by performing a network mapping operation over a first set of ports to generate a list of hosts that are found to be alive; performing a sequential scan on only those hosts in the list to detect the operating system of each host in the list; and performing a parallel port scan to detect characteristics of only those hosts in the list.

In yet another aspect of the embodiments of this invention there is a system that comprises at least one computer coupled with at least one memory containing computer-executable program code stored thereon or therein. The at least one computer is coupled with a data communications network. Execution of the program code results in the system operating a network mapping tool by performing a network mapping operation over a first set of ports to generate a list of hosts that are found to be alive; performing a sequential scan on only those hosts in the list to detect the operating system of each host in the list; and performing a parallel port scan to detect characteristics of only those hosts in the list.

DETAILED DESCRIPTION

The exemplary embodiments of this invention pertain at least in part to improvements in network mapping procedures. While described below primarily in the context of NMAP, those skilled in the art should appreciate that the embodiments of the invention have a wider applicability, and should not be viewed as having utility only in the context of NMAP.

A non-limiting aspect of this invention addresses the issue of the NMAP OS identification inconsistency when using a conventional, default approach. Steps of the method described by this invention include the following.

NMAP is launched with an option to scan a set of IP address ranges, and find a host or hosts “alive”. There is no application discovery at that point. Instead, searching is performed to determine if there are services on the basic OS ports (e.g., tcp, udp, http, win networking (Windows, a registered trademark of Microsoft Corporation, networking)). One potential optimization of this step is to launch NMAP with a “Parallel host scan” option enabled in order to reduce the scanning time. A result of the operation of this step is the generation of a list of hosts alive.

In accordance with the exemplary embodiments of this invention, NMAP is launched again with the options enabled to: (1) scan the IP addresses of the hosts found alive at step1A, and (2) look up all ports for discovering the OS, applications and services. One potential optimization is to launch NMAP with the “Parallel host scan” and “Probe parallelization” options enabled in order to reduce the scanning time. A result of the operation of this step is raw data including OSs, applications and services that are discovered.

The NMAP output is processed to select the relevant information, and to yield the OSs, applications and services in the network or portion of a network of interest.

Significant advantages of using this procedure are a more accurate assessment of an IT environment of interest, as compared to conventional NMAP default scanning, as well as faster discovery since manual involvement required to validate an uncertain OS is minimized.

FIG. 1is a logic flow diagram of the optimized NMAP discovery procedure that was briefly described above. Those skilled in the art will appreciate, however, that the specific IT environment can be replaced with any other IT infrastructure without change to the underlying spirit and scope of the invention.

Step1A: NMAP is launched with the option to scan a set of IP address ranges, and find hosts alive, possibly with the “Parallel host scan” option enabled in order to reduce the scanning time.

A specific embodiment of an NMAP command line to accomplish this step may be:

Note that only a subset of the full set of ports is specified.

Referring to, for example, Chapter 15: Nmap Reference Guide, Options Summary, the general format (usage) is:

and in the example given above the -OG command option specifies OS detection with the output scan in Grepable format sent to file “hostsalive.txt”, and -p specifies a range of specified port IP addresses to scan.

Step1B: NMAP is then launched with the option to scan the IP addresses of the hosts found alive at step1A, and to look up all ports for discovering the OS, applications and services. The option to launch NMAP with the “Parallel host scan” and “Probe parallelization” can be enabled.

A specific embodiment an NMAP command line to accomplish this step may be:

In this case the -O command option specifies OS detection, the -v command option specifies increase verbosity level, -sV specifies probe open ports to determine service/version information, -sS specifies a scan technique of TCP SYN, -p specifies all ports in the range of 1 to 65535, --version-all specifies to try every single probe (intensity 9), and --append-output specifies that the output is to be appended to the previously specified “hostsalive.txt” file, and that the hosts to be scanned are those found in the “hostsalive.txt” file as populated with hosts found alive during the first scan of step1A. The -oX option forces the results to be in XML format to facilitate parsing.

Step1C: Match and Filter:

a) If NMAP is able to detect an OS that matches a fingerprint, report the OS.

b) Examine the results for certain services (such as Microsoft windows networking) that report OS information and if detected, report that as the OS.

c) If there are matches with a high probability of being accurate (e.g., greater than 90%), report all such matches as options to the user.

Beneficial results of the use of this two-step scanning procedure include, but are not limited to, (i) increased accuracy of detection of OSs, applications and services that exist in a particular IT environment and (ii) an acceleration of the discovery process through parallel scanning.

It can be noted that the original list of ports can be determined by being obtained from the potential RMIS customer. For example, the customer can provide the subnets deployed in his environment. Applications that are in common use have fixed ports that serve as the equivalent of ‘well-known ports’, e.g., 50000 for DB2. The scanning application tries all the ports on the machines whose IP belong to the subnet identifier provided by the customer.

It can be further noted that the second scan provides the OS with a certain accuracy, while a third scan can provide services that may imply deterministically a certain OS. In this case the certain OS (if available) is used instead of the OS determined from the second scan.

This third scan can be implemented using a pool, e.g., a fixed size pool, of Java™ threads to manage separate invocations of NMAP during the application scan. More specifically, for all live servers a method scans for running applications by creating a queue of jobs that run NMAP with a specific set of options to perform the detailed scan. The queue of jobs in run directly on the thread pool. The first N jobs start immediately, then as each job in the pool completes the next job in the pool is started.

The pool itself is provided by the Java™ platform. One suitable class for this purpose is: Class ThreadPoolExecutor, found in Java™ 2 Platform, Standard Ed. 5.0.

An aspect of this invention is operating a network mapping tool to perform a network mapping operation over a first set of ports to generate a list of hosts that are found to be alive; to perform a sequential scan on only those hosts in the list to detect the operating system of each host in the list; and to performing a parallel port scan to detect characteristics of only those hosts in the list, such as services and applications associated with the hosts in the list.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method and/or computer program product stored on or in a computer-readable storage medium.

Referring toFIG. 2there is shown an example of a computer or data processor100that is coupled with at least one computer readable medium such as a memory102that stores executable program code104and other information. Also coupled with the computer or data processor100is at least one input/output (I/O) circuit106suitable for receiving data from a network108and for transmitting data to the network. The components shown inFIG. 2may be considered to represent a data processing system110.

In one non-limiting embodiment of this invention the data processing system110can be a part of a RMIS site that provides a management and monitoring service for a remote IT infrastructure environment200that includes one or more hosts (e.g., Host—1, Host—2, . . . , Host_n) that reside behind a FW202, and that are interconnected via an internal network or networks, such as a local area network204. The remote IT infrastructure environment200may be associated with an on-line retailer or any type of for-profit or not-for-profit organization, and can include web servers, application servers, databases and the like. In this case the enhanced network scanning/mapping tool in accordance with the embodiments of this invention can comprise a part104A of the program code104. The enhanced network scanning/mapping tool can be used at least during the initial on-boarding of the remote IT infrastructure environment200as a customer/client of the RMIS provider, and can be employed to rapidly and correctly generate an inventory list of IT resources located in the remote IT infrastructure environment200that are to be managed/monitored by the RMIS provider.

Reference in this regard may be made to commonly owned and copending U.S. patent application Ser. No. 12/729,672, filed Mar. 23, 2010, entitled “Service Method for Customer Self-Service and Rapid On-Boarding for Remote Information Technology Infrastructure Monitoring and Management”, by Anthony Dasari, Michael R. Head, Anca Sailer, Hidayatullah Shaikh, Manu Sharma and Mahesh Viswanathan.

FIG. 3presents a logic flow diagram that is illustrative of the operation of a method, as well as a result of the execution of computer program instructions stored in a tangible memory medium, such as the memory102ofFIG. 2, or provided or conveyed in an intangible form. The operations include, at Block3A, performing a network mapping operation over a first set of ports to generate a list of hosts that are found to be alive, at Block3B, performing a sequential scan on only those hosts in the list to detect the operating system of each host in the list and, at Block3C, performing a parallel port scan to detect characteristics of only those hosts in the list.

As such, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. As but some examples, the use of other similar or equivalent network mappers may be used by those skilled in the art. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention.