Fingerprint determination for network mapping

A system for determining fingerprints includes an interface to receive an indication to determine fingerprints using a set of client data, and a processor to determine a set of indicators based at least in part on the client data and for one or more indicators of the set of indicators, determine whether the indicator comprises a fingerprint based at least in part on a frequency analysis, and in the event it is determined that the indicator comprises a fingerprint, store the fingerprint in a fingerprint database associated with the client.

BACKGROUND OF THE INVENTION

Internet connected assets (e.g., computers, mobile devices, server systems, client systems, internet-of-things devices, etc.) include computing systems in communication with the Internet. Internet connected assets commonly have one or more publicly addressable communication ports, allowing any Internet connected device to query the asset. Some devices allow a range of connection types (e.g., HTTP connections HTTPS connections, FTP connections, FTPS connections, telnet connections, SSH connections, etc.) over the one or more publicly accessible ports. Internet connected assets can be a wide range of different types of hardware devices running a wide range of software including a wide range of configuration options, creating a myriad of possibilities for security vulnerabilities. A typical systems administrator may not be aware of every detail of every system under his or her watch, creating a problem where system vulnerabilities may go undetected and unfixed.

DETAILED DESCRIPTION

A system for determining fingerprints comprises an interface to receive an indication to determine fingerprints using a set of client data, and a processor to determine a set of indicators based at least in part on the client data and for one or more indicators of the set of indicators, determine whether the indicator comprises a fingerprint based at least in part on a frequency analysis, and in the event it is determined that the indicator comprises a fingerprint, store the fingerprint in a fingerprint database associated with the client. In some embodiments, the system for determining fingerprints additionally comprises a memory coupled to the processor and configured to provide the processor with instructions.

In some embodiments, a system for network mapping using a fingerprint comprises a system for identifying computer systems associated with a client network. The fingerprint comprises an identifying criterion or criteria for a computer system. In some embodiments, the fingerprint comprises a set of publicly available information known to be associated with the client network. In various embodiments, a fingerprint comprises a set of open ports (e.g., a specific set of open ports that have been determined to be correlated with the client network), a set of services and associated ports, a webpage component, a web application and associated set of components, an encryption certificate, a public encryption key, a text string, a text string pattern, a domain name, a host name, a host name pattern, a combination of identifying elements, or any other appropriate fingerprint information. In some embodiments, the system for network mapping maps a client network by exhaustively scanning network data and identifying network data that matches the fingerprint. In some embodiments, the system for network mapping scans data collected by a network scanner and stored in a network database (e.g., a network scanning tool first collects all possible information about the network—e.g., the entire Internet—and stores it in a network information database; the system for network mapping then scans the data in the network information database for systems with stored information that matches the fingerprint). In some embodiments, the system for network mapping scans network addresses on the Internet and analyzes received information for systems that return information that matches the fingerprint. When a system is found that is determined to be part of the client network (e.g., its information matches the fingerprint information), the address associated with the system is stored in a client network database. In some embodiments, other system information is additionally stored in the client network database.

In some embodiments, the system for network mapping additionally comprises a system for fingerprint determination. In some embodiments, the system for fingerprint determination comprises a system for automatically determining a fingerprint or a set of fingerprints that accurately distinguish network systems that are part of the client network from network systems that are not part of the client network. The system for determining a fingerprint comprises an indicator determiner for determining a set of identifiers based at least in part on client data. In some embodiments, identifiers comprise potential fingerprints (e.g., a set of identifying data that can comprise a fingerprint but that has not yet been determined whether it accurately distinguishes network systems that are part of the client network from network systems that are not part of the client network). The indicator determiner scans the client data and selects each element comprising identifier data (e.g., an open port, a service and associated port, a webpage component, a web application and associated set of components, an encryption certificate, a public encryption key, a text string, a text string pattern, a domain name, a host name, a host name pattern, etc.). Each indicator is added to a set of indicators. A determination is then made for one or more indicators of the set of indicators (e.g., all of the indicators, a subset of the indicators) whether or not the indicator comprises a fingerprint. The determination is made based at least in part on a frequency analysis. In some embodiments, the frequency analysis comprises a comparison of the frequency the indicator matches network systems within a known client network with the frequency the indicator matches network systems on the Internet. In some embodiments, in the event the indicator matches network systems of the client at a much higher rate than network systems on the Internet, it is determined that the indicator comprises a fingerprint. In some embodiments, determining that the indicator comprises a fingerprint comprises determining that the indicator matches systems within the client network at greater than a threshold frequency. In some embodiments, determining that the indicator comprises a fingerprint comprises determining that the indicator matches systems on the Internet at less than a threshold frequency.

In various embodiments, indicators are single identifiers, pairs of identifiers, triplets of identifiers, or any other combination of identifiers. Each indicator is analyzed by determining whether the indicator appears frequently in the known client network and infrequently in the internet as a whole. For example, an indicator meets this criterion in the event that the indicator frequency in the known client network is above a client network match threshold and the indicator frequency in the internet as a whole is below an internet match threshold. Indicators meeting this criterion are candidate fingerprints that can be ranked by a score. A candidate fingerprint can be accepted as a fingerprint in the event that the score is above a threshold or in the event that the fingerprint is in the top N number of candidate fingerprints (e.g., a certain number of fingerprints).

In some embodiments, the indicators in the set of indicators are not all analyzed—for example, in the event that initially all combinations of 3 identifiers are analyzed on the internet and within the client's network, but after analysis of a percentage of the set indicators a sufficient number of fingerprints have been selected, then the rest of the set is not analyzed. In some embodiments, the analysis of the set is processed in order of highest score to lowest score, where the score indicates a likelihood of being a fingerprint. For example, a score is calculated based at least in part on the percentile of the frequency of occurrence of the indicator within the client network (percentile_fclient) and the percentile of the frequency of occurrence of the indicator in the internet (percentile_finternet).

FIG. 2is a block diagram illustrating an embodiment of a network system. In some embodiments, network system200comprises a network system ofFIG. 1(e.g., network system106). In the example shown, network system200comprises processor202, data storage204, and network interface206. In some embodiments, network system200comprises an Internet connected asset (e.g., a desktop computer, a laptop computer, a smartphone, a tablet computer, a server system, an Internet-of-things device, or any other appropriate Internet connected asset). In various embodiments, processor202comprises a processor for executing instructions, processing data, responding to commands, etc. In various embodiments, processor202comprises a general-purpose processor, a microcontroller, a parallel processing system, a cluster of processors, or any other appropriate processor. Data storage204comprises a data storage for storing data, for storing instructions for processor202, for storing configuration information, or for storing any other appropriate information. In various embodiments, data storage204comprises one or more of a volatile memory, a non-volatile memory, a magnetic memory, an optical memory, a phase-change memory, a semiconductor memory, a disc memory, a tape memory, or any other appropriate memory. Network interface206comprises a network interface for communicating with a network. In the example shown, network interface206comprises network communications information208and a plurality of ports (e.g., port210). Network communications information208includes network communications software, network communications settings, network communications data, or any other appropriate network communications information. The plurality of ports comprises physical ports (e.g., plugs for connecting cables to network system200) or virtual ports (e.g., virtual communications channels identified by a virtual port number). In some embodiments, network interface206comprises a network address (e.g., a network address assigned by an external network addressing authority). In some embodiments, communication with network system200is specified by indicating the network address of network200along with a port number. In some embodiments, some ports of network interface206are configured for communication (e.g., comprising open ports) and some are configured to not respond to communication. In some embodiments, open port configuration information is stored in network communications information208. In some embodiments, some ports are associated with one or more specific communications services (e.g., hypertext transmission protocol (HTTP), file transfer protocol (FTP), secure shell (SSH), etc.). In some embodiments, configuration information associating services with ports is stored in network communications information208. In some embodiments, network communications information208comprises webpage (e.g., non-rendered webpage comments, organization specific images and GIFs, etc.) and web application and web application component information (e.g., 1stparty and 3rdparty javascript code, API calls, web application configuration and version information, etc.) In some embodiments, network communications information208comprises encryption information (e.g., a public SSH key, a certificate, etc.). In some embodiments, network communications information208comprises a network system name or names (e.g., a hostname, a domain name, a set of hostnames, a hostname pattern, etc.). In some embodiments, network communications information comprises text information associated with a service or a set of services (e.g., a welcome text, a connection refused text, a service not supported text, a file not found text, or any other appropriate text information). In some embodiments, network interface206comprises a set of network hardware (e.g., a modem) running a set of communications software that has been configured according to a set of communications specifications.

FIG. 3is a block diagram illustrating an embodiment of a network mapping system. In some embodiments, network mapping system300comprises network mapping system104ofFIG. 1. In some embodiments, network mapping system300comprises a server system. In the example shown, network mapping system300comprises processor302, data storage304, and network interface306. Processor302comprises a processor for executing instructions, processing data, responding to commands, etc. In various embodiments, processor302comprises a general-purpose processor, a microcontroller, a parallel processing system, a cluster of processors, or any other appropriate processor. Processor302includes network scanner308with software and/or hardware that implements network mapping system functionality. Processor302includes fingerprint determiner310for determining fingerprints for use by network scanner308.

In various embodiments, data storage304comprises a data storage for storing data, for storing instructions for processor302, for storing configuration information, or for storing any other appropriate information. In various embodiments, data storage304comprises one or more of a volatile memory, a non-volatile memory, a magnetic memory, an optical memory, a phase-change memory, a semiconductor memory, a disc memory, a tape memory, or any other appropriate memory. In the example shown, data storage304comprises fingerprint database312for storing fingerprints for identifying network systems. In some embodiments, fingerprints stored in fingerprint database are each associated with a client network. In some embodiments, a fingerprint comprises one or more network identifying characteristics. In some embodiments, network identifying characteristics comprise network communications information settings (e.g., a set of open ports, a set of services and associated ports, a webpage component, a web application and set of web application set of components, encryption information, host name information, domain name information, text information, etc.). In some embodiments, fingerprints stored by fingerprint database312are determined by fingerprint determiner310. Data storage304additionally comprises client network database314for storing client network information (e.g., Internet addresses—for example, IP addresses—associated with the client network, network system information associated with network systems associated with the client network, etc.). In some embodiments, after a network mapping process is executed, client network database314comprises a set of client network information describing the extent of the client network (e.g., identifying all network systems found that are associated with the client network). Data storage304additionally comprises network information database316for storing network information. In some embodiments, network information comprises network information received as a result of scanning a network. In some embodiments, network information comprises responses compiled by scanning the Internet. In some embodiments, scanning the Internet comprises providing a payload (e.g., a predetermined data packet or set of packets) to a set of Internet addresses. In some embodiments, scanning a network comprises collecting network information from a set of network systems. In some embodiments, scanning a network comprises collecting network information from all accessible network systems. In various embodiments, network information comprises network communications information settings, network addresses, information received by interactively querying network systems (e.g. information received by performing a follow-up probe in response to an indication of an active service running on the network system), or any other appropriate network information. In some embodiments, network information is stored remotely (e.g., on a storage server, on a different network system, on cloud storage, etc.). In the example shown, network interface306comprises a network interface for interacting with remote systems via a network. In some embodiments, network interface306comprises a network interface configured for high bandwidth communication.

FIG. 4is a flow diagram illustrating an embodiment of a process for determining fingerprints. In some embodiments, the process ofFIG. 4is executed by a network mapping system (e.g., network mapping system104ofFIG. 1). In the example shown, in400, an indication to determine fingerprints using a set of client data is received. For example, the set of client data comprises a known client network (e.g., a set of addresses of computers known to be part of the client network) or network configuration information (e.g., a set of configuration data associated with computers that are part of the client network). In402, a set of indicators is determined based at least in part on the client data. For example, indicators comprise potential fingerprints (e.g., a set of one or more system identifying characteristics as in a fingerprint, wherein it has not yet been determined whether the indicator is able to distinguish network systems associated with the client from network systems not associated with the client). In some embodiments, the set of indicators comprises the set of network identifying elements determined from the client data. In some embodiments, the set of indicators additionally includes all combinations of network identifying elements determined from the client data (e.g., all pairwise combinations, all combinations of 3 elements, all combinations of 4 elements, etc.). In404, a next indicator of the set of indicators is selected. In some embodiments, the next indicator comprises the first indicator. In406, it is determined whether the indicator comprises a fingerprint based at least in part on a frequency analysis. In some embodiments, determining whether the indicator comprises a fingerprint comprises determining whether the indicator is able to distinguish network systems associated with the client from network systems not associated with the client. In some embodiments, determining whether the indicator comprises a fingerprint comprises comparing a frequency that the indicator is determined to match data associated with systems within a known client network with a frequency that the indicator is determined to match data associated with systems on the Internet. In408, in the event it is determined that the indicator does not comprise a fingerprint, control passes to412. In the event it is determined that the indicator comprises a fingerprint, control passes to410. In410, the fingerprint is stored in a fingerprint database associated with the client. In412, it is determined whether there are more indicators of the set of indicators. In the event it is determined that there are not more indicators of the set of indicators, the process ends. In the event it is determined that there are more indicators of the set of indicators, control passes to414. In414, it is determined whether to analyze more indicators of the set of indicators. In some embodiments, only one or more indicators of the set of indicators (e.g., fewer than the complete set of indicators) are analyzed. In some embodiments, all indicators of the set of indicators are analyzed. In various embodiments, it is determined to stop analyzing indicators after a particular level of combinations is analyzed (e.g., after combinations of 3 elements are analyzed), after a measured frequency of occurrences within the known client network drops below a threshold (e.g., after it is determined that all combinations of 4 elements match fewer than a threshold fraction of network systems within the known client network), after a predetermined number of fingerprints are determined, or it is determined to stop analyzing indicators for any other appropriate reason. In the event it is determined not to analyze more indicators, the process ends. In the event it is determined to analyze more indicators, control passes to404.

FIG. 5is a flow diagram illustrating an embodiment of a process for determining a set of indicators based at least in part on client data. In some embodiments, the process ofFIG. 5is used to implement402ofFIG. 4. The process ofFIG. 5is for determining a set of indicators based at least in part on client data of a known client network. In the example shown, in500, a next client system of the known client network is selected. In some embodiments, the next client system comprises the first client system. In502, the client system is queried for network communications information settings. In various embodiments, network communications information settings comprise system configuration files, system configuration scripts, system description files, system responses to one or more communications queries (e.g., a telnet query, a ftp query, a ping, etc.), or any other appropriate network communications information settings. In504, open ports (e.g., ports that respond to a communications request) indicated by the network communications information settings are determined. In506, services and associated ports (e.g., network services determined to be accessible via an associated port) indicated by the network communications settings are determined. In508, certificates indicated by the network communications information settings are determined. In509, webpage components (e.g., non-rendered comments in the webpage, 3rdparty API calls, etc.) indicated by the network communications information settings are determined. In510, web application and web application components (e.g., application version and configuration information, 3rdparty application set of components, etc.) indicated by the network communications information settings are determined. In511, public encryption keys indicated by the network communications information settings are determined. In512, text patterns indicated by the network communications information settings are determined. In some embodiments, text patterns comprise text patterns determined not to be default text patterns (e.g., text patterns produced by an unmodified configuration of a network service). In some embodiments, text patterns comprise text patterns determined to be associated with a client (e.g., a client name, a modification of a client name, a client domain name, a client server name, a client motto, a client advertising phrase, a client internally used phrase, or any other appropriate client associated phrase). In514, the determined information (e.g., information determined in504,506,508,510, and512) is added to a set of indicators. In516, combinations of indicators (e.g., pairwise combinations of indicators of the set of indicators, combinations of 3 indicators of the set of indicators, combinations of 4 indicators of the set of indicators, etc.) are added to the set of indicators. In518, it is determined whether there are more client systems (e.g., in the known client network). In the event it is determined that there are more client systems in the client network, control passes to500. In the event it is determined that there are not more client systems in the client network, the process ends.

FIG. 6is a flow diagram illustrating an embodiment of a process for determining a set of indicators based at least in part on client data. In some embodiments, the process ofFIG. 6is used to implement402ofFIG. 4. The process ofFIG. 6is for determining a set of indicators based at least in part on client data of network configuration information (e.g., network configuration information provided by a client, a client system administrator, etc.). In the example shown, in600, open ports indicated by the client network configuration are determined. In602, services and associated ports indicated by the client network configuration are determined. In604, certificates indicated by the client network configuration are determined. In605, webpage components indicated by the client network configuration are determined. In606, web application and web application components indicated by the client network configuration are determined. In607, public encryption keys indicated by the client network configuration are determined. In608, text patterns indicated by the client network configuration are determined. In610, determined information (e.g., information determined in602,604,605,606,607,608, and610) is added to a set of indicators. In612, combinations of indicators (e.g., pairwise combinations of indicators of the set of indicators, combinations of 3 indicators of the set of indicators, combinations of 4 indicators of the set of indicators, etc.) are added to the set of indicators.

FIG. 7is a flow diagram illustrating an embodiment of a process for determining whether an indicator comprises a fingerprint based at least in part on a frequency analysis. In some embodiments, the process ofFIG. 7implements406ofFIG. 4. In the example shown, in700, a description of a known client network is received. In some embodiments, a description of a known client network comprises a set of addresses of network systems known to be associated with the client network. In702, a next network system of the known client network is selected. In some embodiments, the next network system comprises the first network system. In704, network communications information for the selected network system is requested. In some embodiments, requesting network communications information for the selected network system comprises communicating with the network system to request network communications information. In some embodiments, network communications information is received in response to the request. In706, it is determined whether the network communications information matches the indicator (e.g., whether network communications information received in response to the request of704comprises information matching the network communications information indicated by the indicator). In708, it is determined whether there are more network systems (e.g., of the known client network). In the event it is determined that there are more network systems, control passes to702. In the event it is determined that there are not more network systems, control passes to710. In710, a next network system on the Internet is selected. In some embodiments, the next network system on the Internet comprises the first network system on the Internet. In some embodiments, the network systems on the Internet comprise systems sorted by IP address (e.g., the first network system on the Internet comprises the system with the IP address 0.0.0.0 and the last network system on the Internet comprises the system with the IP address 255.255.255.255). In712, network communication information for the selected network system is requested. In some embodiments, requesting network communications information for the selected network system comprises communicating with the selected network system to request the network communications information. In some embodiments, requesting network communications information for the selected network system comprises requesting stored network communications information (e.g., network communications information stored in a network information database). In714, it is determined whether the network communications information matches the indicator. In716, it is determined whether there are more network systems (e.g., on the Internet). In the event it is determined that there are more network systems, control passes to710. In the event it is determined that there are not more network systems, control passes to718. In718, the frequency the indicator matches network systems of the known client network and the frequency the indicator matches network systems of the Internet are determined. In720, it is determined whether the indicator comprises a fingerprint. In some embodiments, it is determined whether the indicator comprises a fingerprint based at least in part on a frequency analysis. In various embodiments, determining whether the indicator comprises a fingerprint comprises determining whether the frequency the indicator matches network systems of the known client network is greater than the frequency the indicator matches network systems of the Internet, determining whether the frequency the indicator matches network systems of the known client network is much greater than the frequency the indicator matches network systems of the Internet (e.g., is greater by at least a threshold amount), determining whether the frequency the indicator matches network systems of the known client network is greater than a threshold, determining whether the frequency the indicator matches network systems of the Internet is less than a threshold, or in any other appropriate way.

FIG. 8is a flow diagram illustrating an embodiment of a process for determining whether a response matches an indicator. In some embodiments, the process ofFIG. 8comprises a flow diagram for determining whether a response matches a fingerprint. In some embodiments, the process ofFIG. 8implements706ofFIG. 7 and 714ofFIG. 7. In the example shown, in800, a next criterion is selected. In some embodiments, the next criterion comprises the first criterion. In802, the criterion type is determined. In various embodiments, the criterion type comprises a set of open ports criterion type, a set of services and associated ports criterion type, a webpage component criterion type, a web application and associated set of web application components criterion type, a certificate criterion type, a public encryption key criterion type, a text string criterion type, a text pattern criterion type, a domain name criterion type, a host name criterion type, a host name criterion type, or any other appropriate criterion type. In804, it is determined whether the response matches the criterion for the criterion type. In the event it is determined that the response does not match the criterion for the criterion type, control passes to806. In806, the process indicates that the response does not match the indicator, and the process ends. In the event it is determined that the response matches the criterion for the criterion type in804, control passes to808. In808, it is determined whether there are more criteria. In the event it is determined that there are more criteria, control passes to800. In the event it is determined that there are not more criteria, control passes to810. In810, the process indicates that the response matches the indicator. In the example shown, the response is determined to match the indicator only in the event that the response is determined to match all criteria.

FIG. 9is flow diagram illustrating an embodiment of a process for determining whether an indicator comprises a fingerprint. In some embodiments, the process ofFIG. 9is used to implement720ofFIG. 7. In the example shown, in900it is determined whether an indicator match to a client network is greater than a client network match threshold. For example, the indicator is associated with X % of the IP addresses of the client network and X % is greater than a client network match threshold of Y %. In the event that the indicator match to a client network is not greater than a client network match threshold, then the process ends. In the event that indicator match to a client network is greater than a client network match threshold, then control passes to902. In902, it is determined whether an indicator match to the Internet is less than an internet match threshold. For example, the indicator is associated with Z % of the IP addresses of the internet and Z % is greater than an internet match threshold of W %. In the event that the indicator match to the Internet is not less than an internet match threshold, then the process ends. In the event that that the indicator match to the Internet is less than an internet match threshold, then control passes to904.

In904, a score is determined for the indicator to rank the indicator in the set of indicators. For example, a score is determined indicating likelihood of the indicator being a fingerprint. In various embodiments, the score is proportional to one or more of the following: the percentile of the frequency of occurrence of the indicator within the client network (percentile_fclient), the percentile of the frequency of occurrence of the indicator in the internet (percentile_finternet), percentile_fclient×percentile_fclient, percentile_fclient+percentile_fclient, the rank of the frequency of occurrence of the indicator within the client network (rank_fclient), the rank of the frequency of occurrence of the indicator in the internet (rank_finternet), rank_fclient×rank_fclient, rank_fclient+rank_fclient, or any other appropriate factor.

In906, it is determined whether the rank of the indicator within the set of indicators is above a fingerprint threshold. For example, the score is used for each indicator to rank the indicators and it is determined whether the rank of the indicator is above a fingerprint threshold. In the event that the rank of the indicator within the set of indicators is not above a fingerprint threshold, then the process ends. In the event that the rank of the indicator within the set of indicators is above a fingerprint threshold, then in908it is indicated that the indicator comprises a fingerprint. For example, the indicator is flagged as a fingerprint and stored in a fingerprint database.