Patent Description:
Flow log databases store a record of network traffic processing events that occur on a computer network. For example, the network appliances providing network services for a network can create a log entry for each network packet received and for each network packet transmitted. Flow log databases in data centers can be extremely large due to the immense amount of network traffic. General purpose databases, such as Elastic or Lucene, have been used for the flow log databases in some large data centers. At webpage https://www. com/blog/indexing-and-searching-vpc-flow-logs-in-ibm-cloud-databases-for-elasticsearch/, a post discloses a configuration and a sample of a code used to index VPC Flow Logs to the search and analytics engine "Elasticsearch".

The following presents a summary of one or more aspects of the present disclosure, in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated features of the disclosure and is intended neither to identify key or critical elements of all aspects of the disclosure nor to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in a form as a prelude to the more detailed description that is presented later.

The invention will become more fully understood upon a review of the detailed description, which follows. Other aspects, features, and embodiments will become apparent to those of ordinary skill in the art, upon reviewing the following description of specific, exemplary embodiments in conjunction with the accompanying figures. While features may be discussed relative to certain embodiments and figures below, all embodiments can include one or more of the advantageous features discussed herein. In other words, while one or more embodiments may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with the various embodiments discussed herein. In similar fashion, while exemplary embodiments may be discussed below as device, system, or method embodiments such exemplary embodiments can be implemented in various devices, systems, and methods.

The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by this detailed description.

Network traffic flow logs have proven useful in monitoring computer networks, detecting network traffic patterns, detecting anomalies such as compromised (hacked) systems, and for other purposes. Network traffic, however, is increasing. Data centers in particular have enormous amounts of network traffic. Per device flow logs have been collected, processed and stored in general purpose data stores such as Elastic and Lucene. Elasticsearch, based on the Lucene library, provides a distributed, multitenant-capable full-text search engine with a hypertext transport protocol (HTTP) web interface and schema-free java script object notation (JSON) documents. As such, the per device flow logs can be translated into JSON documents and stored. Lucene and Elastic are powerful. They are also general purpose and require massive amounts of storage and processing when storing and searching massive databases. Logging network traffic flows generates massive amounts of data. A more specialized data store has been developed that more efficiently provides searchable logs of network traffic.

The data store can center on an object store. Object stores are often used for data storage applications. Network appliances such as routers, switches, and network interface cards (NICs) can record log entries for a predetermined time period, until available logging storage fills, etc. The log entries can be comma-separated variable (CSV) files with each line being a CSV list of data fields such as source IP, destination IP, etc. The object store can receive the CSV files and process them to produce index objects and flow log objects. The data stored in object store can be in a highly compressed format. Object data and flow log data stored in memory shards can be also be stored in a compressed form optimized for concurrent reading and writing. The compression techniques used for memory shards and on-disk shards can be different depending on the decompression speed needed for searching. The flow log objects can store the log entries while the index object contains tables that can be used for finding specific log entries based on indexed fields of the log entries. The object data store being distributed, the index object can be sharded such that some tables can be searched in parallel. In some embodiments, the index object and the flow log object are contained in a self-indexed searchable object. A file format, disclosed herein, for self-indexed searchable objects is provided that can be rapidly searched, particularly when the object store keeps different parts of the index in different shards. The self-indexed searchable object has been optimized for the specific use case of storing and searching for log entries of network traffic flows.

One advantage is that the data being added to the data store can be added as append only data because neither the index nor the log entries should be modified once stored. This increases the efficiency with which the data can be stored and accessed because the data modification use case does not need to be handled. Another advantage is that the data can be stored in a compact format because the data is not rearranged after being written. For example, the well-known Protobuf utility may be used to serialize and compress the data in the flow log object or in the index object. The index is designed with shards such that only the desired shard must be read from the object store. Another advantage is that searches fail fast because the indexes quickly reveal if the data matching a search query exists. Yet another advantage is the self-indexed searchable object can be stored as flat files with index data and log entries read via read instructions using file offsets and read lengths. Such a mechanism is faster and more efficient than two-layer approaches such as Lucene and Elastic.

The embodiments can have further advantages, as detailed in the following table:.

Network connection flows are flows that can logged as log entries in a flow log object. The contents of a log entry can describe a network connection flow. For example, the fields in a log entry can include: source virtual routing and forwarding (svrf), destination virtual routing and forwarding (dvrf), source IP address (sip), destination IP address (dip), timestamp, source port (sport), destination port (dport), protocol, action, direction, rule identifier (ruleid), session identifier (sessionid), session state, ICMP type (icmptype), ICMP identifier (icmpid), ICMP code (icmpcode), application identifier (appid), forward flow bytes (iflowbytes), and reverse flow bytes (rflowbytes). A log entry can be a comma separated variable (CSV) list of the fields. The definition of the log entry can be the order of the fields in the CSV list. For example, the definition can be {svrf, dvrf, sip, dip, sport, dport, protocol, action, direction, ruleid, sessionid, session state, icmptype, icmpid, icmpcode, appid, iflowbytes, rflowbytes}. A log entry based on the foregoing definition could be {<NUM>, <NUM>, <NUM>. <NUM>, <NUM>. <NUM>, <NUM>, <NUM>, TCP, allow, from-host, <NUM>, <NUM>, dummy, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>}.

<FIG> is a functional block diagram illustrating an object store <NUM> receiving and processing log objects <NUM> according to some aspects. Network appliances <NUM> are receiving and forwarding network packets. The network appliances <NUM> save log entries for the network packets in the log objects <NUM>, which can each contain thousands of log entries. The object store can have a volatile memory <NUM> and a nonvolatile memory <NUM>. Examples of volatile memory include synchronous dynamic random-access memory (SDRAM) such as double data rate <NUM> (DDR4) SDRAM and double data rate <NUM> (DDR5) SDRAM. Examples of nonvolatile memory include hard disk drives, and solid-state drives. Some embodiments may have an object store That is used only for storing the flow log objects, index objects and the log objects reported by the network devices. The processing needed for creating the flow log objects, creating index objects, and performing searches can be performed by a separate service. The separate service can use the object store for storing objects and other data.

In-memory data can be stored in volatile memory <NUM> while persistent data can be stored in nonvolatile memory. The in-memory data can include a memory flow log object <NUM> and a memory index object <NUM>. The memory flow log object <NUM> can include log entries. The memory index object can include a memory flow table <NUM>, memory shards <NUM>, and a memory shards table <NUM>. The memory shards <NUM> can include a first memory shard <NUM>, and a second memory shard <NUM>. The first memory shard <NUM> can include first memory shard entries <NUM>. The second memory shard <NUM> can include second memory shard entries <NUM>. The memory shard entries <NUM>, <NUM> can be flow keys stored in association with memory flow entry indicators. In some embodiments, the flow keys and the memory flow entry indicators are stored as the key-value pairs of a key value data store.

The memory shards table <NUM> can indicate the locations of the memory shards <NUM>. The number of shards is predetermined and may be application specific. For example, an embodiment may have <NUM> memory shards. As such, the memory shards table <NUM> can be a list of <NUM> addresses, offsets, or other location indicators. Each location indicator in the memory shards table indicates the location of a memory shard.

An intake process can be triggered when a log object is received. The intake process can process the log entries in the log object by storing the log entries in the memory flow log object <NUM> and by creating index data stored in the memory index object <NUM>. The index data can include flow entries stored in the memory flow table <NUM> and memory shard entries. The data in the volatile memory <NUM> can be transferred to the nonvolatile memory <NUM> at regular intervals, when a timer expires, or when a threshold number of log entries are stored in the memory flow log object <NUM>. The data in the nonvolatile memory can include flow log object <NUM>, flow series objects <NUM>, and an index object <NUM>. The index object <NUM> can include a flow table <NUM>, shards <NUM>, and a shards table <NUM>. The shards <NUM> can include a first shard <NUM> and a second shard <NUM>. Here, two shards are used in order to explain aspects of the embodiments. It is understood that in practice more shards can be used. Embodiments having sixty-three shards have been tested.

The log entries in the memory flow log object <NUM> can be moved into the flow log object <NUM>. The log entries can be compressed using any of the well-known data compression algorithms. The first shard <NUM> can include first shard entries <NUM>. The second shard <NUM> can include second shard entries <NUM>. The shard entries <NUM>, <NUM> can include flow keys stored in association with flow entry indicators. In some embodiments, the flow keys and the flow entry indicators are stored as the key-value pairs of a key value data store.

The shards table <NUM> can indicate the locations of the shards <NUM>. The number of shards can be the same as the number of memory shards. For example, an embodiment may have <NUM> memory shards and <NUM> shards. As such, the shards table <NUM> can be a list of <NUM> addresses, offsets, or other location indicators. Each location indicator in the shards table indicates the location of a shard.

Flow series objects <NUM> can also be stored in the nonvolatile memory <NUM>. Flow series objects <NUM> can include a first flow series object <NUM>, a second flow series object, a third flow series object, and many more flow series objects. The flow series objects <NUM> record which of the flow log objects and index objects were created during specific time periods. For example, the first flow series object <NUM> may be associated with the time interval from 1AM to 2AM GMT on December <NUM>, <NUM>. As such, the first flow series object <NUM> can be a list of flow log object identifiers that includes a first flow log object identifier <NUM>, a second flow log object identifier, a third flow log object identifier, and many more flow log object identifiers for flow log objects created between 1AM and 2AM GMT on December <NUM>, <NUM>. The first flow log object identifier is illustrated as indicating the index object <NUM>. In some embodiments, a single file, called a flow file, is an internally indexed searchable object that includes the flow log object and the index object. In such embodiments, the flow series objects can be files, called series files, that include a list of file names of flow files. As each new flow file is created, its file name can be appended to the currently open series file. Furthering the example, the currently open series file can be closed at the end of every hour and a new one opened such that each series file is associated with a specific one hour time period and includes the file names of flow files created during that specific one hour time period.

<FIG> is a high-level diagram illustrating processing of a log object <NUM> according to some aspects. The log object <NUM> includes log entries such as a first log entry <NUM>, a second log entry <NUM>, a third log entry <NUM>, and a fourth log entry <NUM>. The log object can be a CSV file and each of the log entries can be a line in the file and can contain a CSV list of field values. In some embodiments, the first line of the file is a CSV list of field names. Those practiced in processing computer data are familiar with CSV files. The first log entry <NUM> is illustrated as a flow log entry generated by a network appliance. The first flow log entry <NUM> includes data fields such as a first field <NUM>, a second field <NUM>, a third field <NUM>, a fourth field <NUM>, a fifth field <NUM>, a sixth field <NUM>, a seventh field <NUM>, an eighth field <NUM>, and a ninth field <NUM>. The first field <NUM> can contain a value indicating the source IP address of a network packet. The second field <NUM> can contain a value indicating the destination IP address of the network packet. The third field <NUM> can contain a value indicating the virtual private cloud tag of the network packet. The fourth field <NUM> can contain a value indicating the entry source Id of the network packet. The fifth field <NUM> can contain a value indicating the source virtual routing and forwarding (VRF) identifier of the network packet. The sixth field <NUM> can contain a value indicating destination VRF identifier of the network packet. The seventh field <NUM> can contain a value indicating the protocol (e.g., layer <NUM> protocol) of the network packet. The eighth field <NUM> can contain a value indicating the source port of the network packet. The ninth field <NUM> can contain a value indicating the destination port of the network packet.

Some of the data fields are indexed fields that include indexed field values. <FIG> shows that the indexed fields are the first field <NUM>, the second field <NUM>, the third field <NUM>, and the fourth field <NUM>. In a non-limiting example, the first field is an indexed field containing the indexed field value <NUM>. <NUM> while the second field is an indexed field containing the indexed field value <NUM>. Each of the indexed fields can be used to determine a shard identifier and a flow key. Flow keys, shard identifiers, and other values may be determined via a hashing algorithm (e.g., CRC <NUM>), via a lookup table, or using some other technique. Note: a modulo <NUM> operation can produce a shard identifier in the range <NUM>-<NUM>.

The first log entry <NUM> can be added (e.g., appended) to the flow object. As such, the first log entry's location is known. For the first field value, a log entry indicator indicating the first log entry's location is added to flow entry "W" in the flow table <NUM>. The first field value <NUM> can be used to determine flow key "A" <NUM>, and shard identifier "B" <NUM>. If not already present, an entry for flow key "A" can be added to shard "B". The shard entry associates flow key "A" with flow entry "W". For the second field value, a log entry indicator indicating the first log entry's location is added to flow entry "X" in the flow table <NUM>. The second field value <NUM> can be used to determine flow key "C" <NUM>, and shard identifier "D" <NUM>. If not already present, an entry for flow key "C" can be added to shard "D". The shard entry associates flow key "C" with flow entry "X". For the third field value, a log entry indicator indicating the first log entry's location is added to flow entry "Y" in the flow table <NUM>. The third field value <NUM> can be used to determine flow key "E" <NUM>, and shard identifier "F" <NUM>. If not already present, an entry for flow key "E" can be added to shard "F". The shard entry associates flow key "E" with flow entry "Y". For the fourth field value, a log entry indicator indicating the first log entry's location is added to flow entry "Z" in the flow table <NUM>. The fourth field value <NUM> can be used to determine flow key "G" <NUM>, and shard identifier "H" <NUM>. If not already present, an entry for flow key "G" can be added to shard "H". The shard entry associates flow key "G" with flow entry "Z".

<FIG> illustrates an example having four indexed fields. It is understood that in practice more or fewer indexed fields may be used. Another example, that has been under test, has the following indexed fields: source IP, destination IP, the dyad < source IP, destination IP >, network appliance identifier, virtual private cloud name, source port, destination port, and protocol.

<FIG> is a high-level block diagram illustrating an internally indexed searchable object <NUM> according to some aspects. The internally indexed searchable object <NUM> includes an index object <NUM> and a flow log object <NUM>. The flow log object includes log entries such as the first log entry <NUM> illustrated in <FIG>. The index object <NUM> includes a shards table <NUM>, shards <NUM>, and a flow table <NUM>. The shards table <NUM> includes shard table entries such as a first shard table entry <NUM>, a second shard table entry <NUM>, and a third shard table entry <NUM>. The shard table entries store shard indicators, such as the first shard indicator <NUM>, that can indicate the location and size of individual shards. The first shard indicator <NUM> can include a first shard location <NUM> and a first shard size <NUM>. An indicator that includes a location and a size can be used for reading the indicated item directly from a memory or a file without searching. The shard indicators may be stored in association with shard identifiers. For example, the first shard indicator <NUM> may be stored in association with a first shard identifier <NUM>. Embodiments using shard identifiers numbered, for example, from <NUM> to <NUM> may simply store the shard indicator for shard N at location N in the table.

Shards <NUM> includes shards such as a first shard <NUM>, a second shard <NUM>, and a third shard. The shards <NUM> include shard entries that store flow entry indicators in association with flow keys. The flow entry indicators can indicate the location and size of flow entries in the flow table <NUM>. The first shard's first entry <NUM> stores a first flow entry indicator <NUM> in association with a first flow key <NUM>. The second shard's first entry <NUM> stores a second flow entry indicator <NUM> in association with a second flow key <NUM>. The first flow entry indicator <NUM> can include a flow entry offset <NUM> and a flow entry size <NUM> that can be used for reading a flow entry from a memory or file. The first flow entry indicator <NUM> is shown indicating a first flow entry <NUM>. The second flow entry indicator <NUM> is shown indicating a second flow entry <NUM>.

The flow table <NUM> includes flow entries such as the first flow entry <NUM> and the second flow entry <NUM>. The flow entries can indicate log entries in the flow log object <NUM>. The first flow entry <NUM> includes log entry indicator <NUM>,<NUM><NUM>, log entry indicator <NUM>,<NUM>, and log entry indicator <NUM>,<NUM>. Log entry indicator <NUM>,<NUM><NUM> includes a log entry offset <NUM> and a log entry size <NUM>. The second flow entry <NUM> includes log entry indicator <NUM>,<NUM>, and log entry indicator <NUM>,<NUM>. Log entry indicator <NUM>,<NUM> and log entry indicator <NUM>,<NUM> are shown indicating the same log entry.

<FIG> is a functional block diagram of a network appliance <NUM> such as a network interface card (NIC) or a network switch having an application specific integrated circuit (ASIC) <NUM>, according to some aspects. A network appliance that is a NIC includes a PCIe connection <NUM> and can be installed in a host computer. A NIC can provide network services to the host computer and to virtual machines (VMs) running on the host computer. The network appliance <NUM> includes an off-ASIC memory <NUM>, and ethernet ports <NUM>. The off-ASIC memory <NUM> can be one of the widely available memory modules or chips such as DDR4 SDRAM modules such that the ASIC has access to many gigabytes of memory. The ethernet ports <NUM> provide physical connectivity to a computer network such as the internet.

The ASIC <NUM> is a semiconductor chip having many core circuits interconnected by an on-chip communications fabric, sometimes called a network on a chip (NOC) <NUM>. NOCs are often implementations of standardized communications fabrics such as the widely used AXI bus. The ASIC's core circuits can include a PCIe interface <NUM>, CPU cores <NUM>, P4 packet processing pipeline <NUM> elements, memory interface <NUM>, on ASIC memory (e.g., SRAM) <NUM>, service processing offloads <NUM>, a packet buffer <NUM>, extended packet processing pipeline <NUM>, and packet ingress/egress circuits <NUM>. A PCIe interface <NUM> can be used to communicate with a host computer via the PCIe connection <NUM>. The CPU cores <NUM> can include numerous CPU cores such as CPU <NUM><NUM>, CPU <NUM><NUM>, and CPU <NUM><NUM>. The P4 packet processing pipeline <NUM> can include a pipeline ingress circuit <NUM>, a parser circuit <NUM>, match-action units <NUM>, a deparser circuit <NUM>, and a pipeline egress circuit <NUM>. The service processing offloads <NUM> are circuits implementing functions that the ASIC uses so often that the designer has chosen to provide hardware for offloading those functions from the CPUs. The service processing offloads can include a compression circuit <NUM>, decompression circuit <NUM>, a crypto/PKA circuit <NUM>, and a CRC calculation circuit <NUM>. The specific core circuits implemented within the non-limiting example of ASIC <NUM> have been selected such that the ASIC implements many, perhaps all, of the functionality of an InfiniBand channel adapter, of an NVMe card, and of a network appliance that processes network traffic flows carried by IP (internet protocol) packets.

The P4 packet processing pipeline <NUM> is a specialized set of elements for processing network packets such as IP packets, NVMe protocol data units (PDUs), and InfiniBand PDUs. The P4 pipeline can be configured using a domain-specific language. The concept of a domain-specific language for programming protocol-independent packet processors, known simply as "P4," has developed as a way to provide some flexibility at the data plane of a network appliance. The P4 domain-specific language for programming the data plane of network appliances is defined in the "<NPL>. P4 (also referred to herein as the "P4 specification," the "P4 language," and the "P4 program") is designed to be implementable on a large variety of targets including network switches, network routers, programmable NICs, software switches, FPGAs, and ASICs. As described in the P4 specification, the primary abstractions provided by the P4 language relate to header types, parsers, tables, actions, match-action units, control flow, extern objects, user-defined metadata, and intrinsic metadata.

The network appliance <NUM> can include a memory <NUM> for running Linux or some other operating system. The memory <NUM> can also be used for assembling a log object <NUM>. The network appliance can be viewed as having a control plane and a data plane. The control plane, sometimes called the slow data path or slow plane, can configure the data plane, sometimes called the fast data path or fast plane, to process network packets. The data plane can be configured via downloaded P4 programming and forwarding data. Once configured for a particular network traffic flow, the data plane can process the network packets for that network traffic flow at line speed and without the control plane's involvement. The data plane can also be configured to store log entries in the log object <NUM> to thereby create a log of the packets that have been processed.

A well configured packet processing pipeline can store log entries while processing network packets at line speed. As such, the log object can get very large very quickly. The network appliance can therefore, at regular intervals, create a new log object to hold new log entries, and can free up the memory used by the previous log object after sending the previous log object to an object store.

The CPU cores <NUM> can be general purpose processor cores, such as ARM processor cores, MIPS processor cores, and/or x86 processor cores, as is known in the field. Each CPU core can include a memory interface, an ALU, a register bank, an instruction fetch unit, and an instruction decoder, which are configured to execute instructions independently of the other CPU cores. The CPU cores may be Reduced Instruction Set Computers (RISC) CPU cores that are programmable using a general-purpose programming language such as C.

The CPU cores <NUM> can also include a bus interface, internal memory, and a memory management unit (MMU) and/or memory protection unit. For example, the CPU cores may include internal cache, e.g., L1 cache and/or L2 cache, and/or may have access to nearby L2 and/or L3 cache. Each CPU core may include core-specific L1 cache, including instruction-cache and data-cache and L2 cache that is specific to each CPU core or shared amongst a small number of CPU cores. L3 cache may also be available to the CPU cores.

There may be multiple CPU cores <NUM> available for control plane functions and for implementing aspects of a slow data path that includes software implemented packet processing functions. The CPU cores may be used to implement discrete packet processing operations such as L7 applications (e.g., HTTP load balancing, L7 firewalling, and/or L7 telemetry), certain InfiniBand channel adapter functions, flow table insertion or table management events, connection setup/management, multicast group join, deep packet inspection (DPI) (e.g., URL inspection), storage volume management (e.g., NVMe volume setup and/or management), encryption, decryption, compression, and decompression, which may not be readily implementable through a domain-specific language such as P4, in a manner that provides fast path performance as is expected of data plane processing.

The packet buffer <NUM> can act as a central on-chip packet switch that delivers packets from the network interfaces <NUM> to packet processing elements of the data plane and vice-versa. The packet processing elements can include a slow data path implemented in software and a fast data path implemented by packet processing circuitry <NUM>, <NUM>.

The packet processing circuitry <NUM>, <NUM> can be a specialized circuit or part of a specialized circuit using one or more ASICs or FPGAs to implement programmable packet processing pipelines. Some embodiments include ASICs or FPGAs implementing a P4 pipeline as a fast data path within the network appliance. The fast data path is called the fast data path because it processes packets faster than a slow data path that can also be implemented within the network appliance. An example of a slow data path is a software implemented data path wherein the CPU cores <NUM> and memory <NUM> are configured via software to implement a slow data path.

The ASIC <NUM> is illustrated with a P4 packet processing pipeline <NUM> and an extended packet processing pipeline <NUM>. The extended packet processing pipeline is a packet processing pipeline that has a direct memory access (DMA) output stage <NUM>. The extended packet processing pipeline has match-action units <NUM> that can be arranged as a match-action pipeline. The extended packet processing pipeline has a pipeline input stage <NUM> that can receive packet header vectors (PHVs) or directives to perform operations. A PHV can contain data parsed from the header and body of a network packet by the parser <NUM>.

All memory transactions in the NIC <NUM>, including host memory transactions, on board memory transactions, and registers reads/writes may be performed via a coherent interconnect <NUM>. In one non-limiting example, the coherent interconnect can be provided by a network on a chip (NOC) "IP core". Semiconductor chip designers may license and use prequalified IP cores within their designs. Prequalified IP cores may be available from third parties for inclusion in chips produced using certain semiconductor fabrication processes. A number of vendors provide NOC IP cores. The NOC may provide cache coherent interconnect between the NOC masters, including the packet processing pipeline circuits <NUM>, <NUM>, CPU cores <NUM>, memory interface <NUM>, and PCIe interface <NUM>. The interconnect may distribute memory transactions across a plurality of memory interfaces using a programmable hash algorithm. All traffic targeting the memory may be stored in a NOC cache (e.g., <NUM> MB cache). The NOC cache may be kept coherent with the CPU core caches.

<FIG> illustrates a flow file <NUM> that is an internally searchable object according to some aspects. The flow file can be created by sequentially writing data into the file, starting with the file header <NUM>. The file header can have a fixed size, such as <NUM> kB and can include a file version identifier. The file version number can be used by programs reading the file for determining if and how to read the file. The flow log object <NUM> can be stored immediately after the flow log header <NUM>. The flow log object <NUM> can be the log entries stored one after another. As discussed, the log entries can be stored in a compressed format. The flow entries of the flow table can be stored immediately after the flow log object <NUM>. The flow entries can be stored one after another beginning with the first flow entry <NUM> and ending with the last flow entry <NUM>. A flow entry can include a number of log entry indicators. The log entry indicators can be log entry offsets paired with log entry sizes. For example, a log entry offset having the value "A" can be paired with the log entry size having the value "B". The corresponding log entry can be read from the flow file <NUM> by reading "B" bytes beginning at location "A" in the flow file <NUM>. The log entries for a flow entry can be read by stepping through its log entry indicators and reading each log entry in turn.

The shards can be stored immediately after the last flow entry <NUM> beginning with the first shard <NUM> and ending with the last shard <NUM>. The shards can be stored as key value pairs. The flowKey can be the key and the log entry indicator can be the value. The log entry indicator can be a flow entry offset and a flow entry size. For example, the flow entry offset can be the value "C" and the flow entry size can be the value "D". The corresponding flow entry can be read from the flow file <NUM> by reading "D" bytes beginning at location "C" in the flow file <NUM>.

A shards table <NUM> can be stored immediately after the last shard <NUM>. The shards table can begin with the first shard table entry and end with the last shard table entry. The shard table entries can include a shard offset and a shard size. For example, the shard offset can be the value "E" and the shard size can be the value "F". The corresponding shard can be read from the flow file <NUM> by reading "F" bytes beginning at location "E" in the flow file <NUM>. The shards table entries can have a known size. For example, each can be eight bytes long and can include a four byte shardOffset and a four byte shardSize. As such, the Nth shard table entry can be read by reading eight bytes beginning at location (N-<NUM>)*<NUM> in the shards table.

A series data object <NUM> can be stored immediately after the shards table. The flow log object <NUM> contains log entries from one or more specific log objects. The log objects can be stored in the object store and the series data object <NUM> can indicate where those specific log objects are stored. For example, the log objects can be stored, perhaps in compressed form, as CSV files within a filesystem. The series data stored in the series data object could include the fully qualified file names of each of those log objects.

The series object data can be series data that is a copy of the content stored in the series file for the time-period represented by the flow log object. The series data can be stored in both the series file and the flow file <NUM> for recovery purposes. For example, if the series file gets lost or gets corrupted then the series can be reconstructed by reading the series data from all the flows files.

A flow log footer <NUM> can be stored immediately after the series data object <NUM>. Flow log footers can all be the same size (e.g., 5kB) and can contain a shards table indicator and a series data object indicator. The shards table indicator can include a shards table offset and a shards table size. For example, the shards table offset can be the value "G" and the shards table size can be the value "H". The shards table can be read from the flow file <NUM> by reading "H" bytes beginning at location "G" in the flow file <NUM>.

Log entries in the flow file <NUM> can be found quickly while reading only the necessary data from the flow file. For example, the entries matching a particular value of an indexed field can be found by determining a flow key and a shard identifier from that value of the indexed field. The flow log footer can be read by reading the tail of the file. The number of bytes to read is known because the size of the footer is known. The shards table can be read using the shards table offset and shards table size from the footer. The shard identifier is used to read, from the shards table, the shard offset and shard size of the shard having the shard identifier. The flow key is used to find a flow entry indicator in the shard. The flow entry indicator indicates a flow entry. The log entry offsets and log entry sizes in the flow entry are used to read the flow entries.

The flow file <NUM> is illustrated as a file that can be stored in a file system on one or more nonvolatile memory devices such as hard drives, solid state drives, etc. In some implementations, the flow file <NUM> can be memory mapped. A file can be memory mapped using a system call such as mmap(), which is a POSIX-compliant Unix system call that maps files or devices into volatile memory such as SDRAM. As such, the contents of the file can be accessed very quickly because the data is already in the system's SDRAM. The flow file <NUM> is well suited for being memory mapped because the desired data fields can be read directly from the SDRAM using known offsets. In addition, the flow file can be read only and, as such, there is additional efficiency because there is no need to synchronize writes from SDRAM to disk. For example, the flow log footer, which has a known size and position, can be accessed using its known position in the file. The flow log footer <NUM> gives the location of the shard table (shardsTableOffset) and the size of the shards table (shardTableSize). As such, the shards table can be accessed directly in SDRAM by accessing shardsTableSize bytes beginning at the location shardsTableOffset in the memory mapped file. Flow log objects and index objects, such as those of <FIG>, may be memory mapped separately and distinctly by, for example, storing them in separate files and memory mapping those files. The shards table location and size parameters, (e.g., shardsTableOffset and shardsTableSize) are illustrated as located at predetermined and specified locations in the footer. The shards table location and size parameters may alternatively be located in the header or in some other location that is predetermined and specified. Furthermore, the data blocks can be ordered differently within the flow file. In fact, the technique of storing data block offsets and data block sizes can be used to intermingle the data blocks.

<FIG> is a high-level flow diagram illustrating a process that can be implemented by a network appliance to produce log objects <NUM> according to some aspects. After the start, at block <NUM> a new log object is created. If the log object is a CSV file, then creating the log object can include creating a new file and writing in a header of field names as a CSV list. It is understood that CSV is just one of the formats may be used. Examples of other formats that may be used include Apache Parquet file format, a proprietary binary object format, etc. At block <NUM>, a logging timer can be set. The logging timer can expire after a logging period (e.g., <NUM> minute) has expired. At block <NUM>, a network packet is received and processed. At block <NUM>, a log entry is created that is a record of the network packet that was received and processed. At block <NUM>, the log entry is stored in the log object. At block <NUM>, the status of the logging timer is checked. If the logging timer has not expired, the process can loop back to block <NUM>. Otherwise, at block <NUM> the network appliance can send the log object to the object store before looping back to block <NUM>. The log object sent to the object store can be a CSV file and the CSV file can be compressed. "Gzip" is a well-known file compression utility that can be used to compress the CSV file. The object store can store the log object.

<FIG> is a high-level flow diagram illustrating the processing of log objects <NUM> according to some aspects. After the start, at block <NUM> the object store can receive a log object. The log object can be uncompressed if needed. At block <NUM>, the header of the log object can be read to confirm which fields are stored in what order. At block <NUM>, the first log entry is read from the log object. At block <NUM>, "log entry" is set to the first log entry. At block <NUM>, log entry is stored at "log entry location" in an in-memory flow log object such as memory flow log object <NUM>. At block <NUM>, the in-memory index object (e.g., memory index object <NUM>) is updated based on the log entry location and the log entry size. At block <NUM>, the entries memory counter is incremented. The entries memory counter can be zeroed out when the in-memory flow log object and the in-memory index object are moved to a persistent memory. At block <NUM>, the entries memory counter is compared to a max memory entries threshold value. For example, the max memory entries threshold value can be set to <NUM>,<NUM>,<NUM> such that no flow log object contains more than <NUM>,<NUM>,<NUM> log entries.

If the max memory entries threshold value is not greater than the entries memory counter the process can proceed to block <NUM>, otherwise it can proceed to block <NUM>. At block <NUM>, the process can persist the in-memory flow log object and the in-memory index object by transferring them to non-volatile memory. In some embodiments the in-memory flow log object and the in-memory index object are written into a flow file such as flow file <NUM>. Persisting the in-memory flow log object and the in-memory index object may be accomplished via a process such as that shown in <FIG>. At block <NUM>, the log object is recorded in a series data object such as the first series data object <NUM>. For example, the log object can be a CSV file and the fully qualified file name of the CSV file can be written in the series data object. At block <NUM> the process can check if more log entries are available in the log object. If not, the process can stop. If more log entries are available in the log object, then at block <NUM> the next log entry is read from the log object. At block <NUM>, "log entry" is set to that next log entry and the process loops back to block <NUM>.

<FIG> is a high-level flow diagram illustrating creation of flow log objects and index objects from in-memory data structures <NUM> according to some aspects. After the start, at block <NUM> a flow log object and an index object can be created as persistent objects in the nonvolatile memory of the object store. At block <NUM>, shards can be created in the index object. As discussed above, the index object can have a set number of shards (e.g., <NUM> shards). At block <NUM>, "current memory shard" is set to the first memory shard and "current object shard" is set to the first shard of the index object. At block <NUM>, the current memory shard is transferred to the current object shard. At block <NUM>, the process can check if the current shard (memory shard or index shard) is the last shard. If not, at block <NUM> the current memory shard is set to the next memory shard and the current object shard is set to the next shard of the index object before the process loops back to block <NUM>. Otherwise, at block <NUM> a flow table can be created in the index object. The data in the in-memory flow table may be used to create the flow table. At block <NUM>, a shards table is created in the index object. At block <NUM>, an internally indexed searchable object can be written to the object store. The internally indexed searchable object can include the flow log object, the index object, and the series data object. At block <NUM>, the in-memory shards, in-memory flow table, and the series data object can be initialized. At block <NUM>, an indicator for the internally indexed searchable object can be recorded in the current flow series object. At block <NUM>, the entries memory counter (checked at block <NUM>) is set to <NUM>. At block <NUM>, the flow log period timer is set to time out after a flow log period such as <NUM> minutes. A <NUM> minute flow log period and a <NUM>,<NUM>,<NUM> max memory entries value can ensure that the in-memory objects are persisted every ten minutes or every six million log entries, whichever occurs first.

<FIG> is a high-level flow diagram illustrating a use of timers while producing index objects and flow log objects <NUM> according to some aspects. After the start, at block <NUM> the flow log period timer can be set to time out after a flow log period such as ten minutes. At block <NUM>, a series period timer can be set to time out after a series period such as one hour. At block <NUM>, a new flow series object can be created. At block <NUM>, the current flow series object is set to the newly created flow series object. Data can be written into the current flow series object at block <NUM> of <FIG>. The process can wait <NUM> for a timer to expire or for a new log object to be received. When a new log object is received then at block <NUM> the new log object can be processed. The process shown in <FIG> may process the new log object. When the flow log period timer times out, the process can proceed to block <NUM>. At block <NUM>, the in-memory index object and the in-memory flow log object can be transferred to persistent objects. The process illustrated in <FIG> may persist the in-memory objects. When the series period timer times out, the process may proceed to block <NUM>. At block <NUM>, a new flow series object is created. At block <NUM>, the current flow series object is set to the newly created flow series object. At block <NUM>, the series period timer is set to timeout after a series period. The process can return to waiting <NUM> after blocks <NUM>, <NUM>, and <NUM>.

<FIG> is a high-level flow diagram illustrating the processing of a search query <NUM> according to some aspects. At block <NUM>, a search query is received. The search query can include values for one or more of the indexed fields. At block <NUM> the process can check if the search query specifies a time period. If so, the process proceeds directly to block <NUM>. Otherwise, at block <NUM> the time period is set to a default time period (e.g., the last <NUM> hours) before the process proceeds to block <NUM>. At block <NUM>, the process determines if the search query includes a value for the first indexed field. If so, at block <NUM> the value for the first indexed field is used to determine a shard identifier and a flow key before the process proceeds to concurrent processing <NUM>. Otherwise, the process proceeds to block <NUM>. At block <NUM>, the process determines if the search query includes a value for the second indexed field. If so, at block <NUM> the value for the second indexed field is used to determine a shard identifier and a flow key before the process proceeds to concurrent processing <NUM>. Otherwise, the process proceeds to block <NUM>. At block <NUM>, the process determines if the search query includes a value for the third indexed field. If so, at block <NUM> the value for the third indexed field is used to determine a shard identifier and a flow key before the process proceeds to concurrent processing <NUM>. Otherwise, the process proceeds to block <NUM>. At block <NUM>, a default value for the third indexed field is used to determine a shard identifier and a flow key before the process proceeds to concurrent processing <NUM>. Note that a default value can be used for any of the indexed fields at block <NUM>. Those practiced in computer programming can extend the example of <FIG> for an arbitrary number of indexed fields.

At concurrent processing <NUM>, the process can split into concurrent threads. One thread can proceed to block <NUM>. At block <NUM>, the flow series objects can be used to identify flow objects or internally indexed searchable objects having entries within the time period being searched. At block <NUM>, the identified flow objects or internally indexed searchable objects can be searched using the shard identifier and the flow key for log entries matching the search query. The threads can rendezvous at block <NUM> where the log entries meeting the search criteria in the search query are returned. Returning to concurrent processing <NUM>, another thread can proceed to block <NUM>. At block <NUM>, the thread determines if the time period includes the present. If not, the thread can exit. Otherwise, at block <NUM> the in-memory index object and the in-memory flow log object can be searched before the rendezvous at block <NUM>. If, at block <NUM>, numerous flow objects or internally indexed searchable objects are identified, then additional threads can be spawned to search them all concurrently.

<FIG> is a high-level flow diagram illustrating using shard identifiers and flow keys for searching an internally indexed searchable object <NUM> according to some aspects. The flow file <NUM> illustrated in <FIG> is an example of an internally indexed searchable object. After the start, at block <NUM> the header, which can be a fixed size, is read. At block <NUM>, the process can determine if the version of the internally indexed searchable object is compatible with the process. If not, at block <NUM> an error is returned and the process exits. Otherwise, at block <NUM> the query shard identifier and the query flow key are determined using the query value. The query value can be the value for an indexed field in a search query. At block <NUM>, the shards table offset and the shards table size can be read from the footer. At block <NUM>, the shards table offset and the shards table size are used to read the shards table. At block <NUM>, the query shard identifier identifies the query shard. The query shard is the shard that includes log entries that match the search query. At block <NUM> the query shard is read. At block <NUM>, the query flow key is used to identify the query flow entry. The query flow entry is the flow entry indicating log entries that match the search query. At block <NUM>, the log entries indicated by the query flow entry are read. At block <NUM>, the matching log entries, which are the log entries matching the search query, can be returned.

<FIG> is a high-level diagram illustrating an internally indexed searchable object <NUM> stored by a distributed object store <NUM> according to some aspects. Network appliances <NUM> can provide log objects to a management node <NUM> that includes a volatile memory <NUM> storing memory shards <NUM>. The management node <NUM> can be one of the nodes participating in a distributed object store <NUM>. The management node <NUM> can process the log objects to create in-memory flow log objects and in-memory index objects. The management node can also transfer the in-memory flow log objects and in-memory index objects to the persistent storage of the object store <NUM>. Numerous nodes can participate in the object store <NUM>. The object store can therefore be distributed across the participating nodes and can be stored as sharded data on the nodes. As such, the internally indexed searchable object <NUM> can be replicated across the nodes.

<FIG> illustrates a high-level flow diagram of a method for a network flow log according to some aspects. At block <NUM>, the process creates a flow log object that includes a plurality of log entries at a plurality of log entry locations. At block <NUM>, the process creates an index object that includes a plurality of shards and a flow table that indicate the log entry locations. At block <NUM>, the process stores the flow log object and the index object in at least one nonvolatile memory, wherein each of the log entries includes a plurality of indexed field values for a plurality of indexed fields that include a first indexed field and a second indexed field, the log entries include a log entry stored at an entry location, a first shard entry is stored in a first one of the shards indicated by a first shard identifier determined using the first indexed field, the first shard entry stores a first flow entry indicator in association with a first flow key determined using the first indexed field, the first flow entry indicator indicates a first log entry indicator that indicates the entry location, a second shard entry is stored in a second one of the shards indicated by a second shard identifier determined using the second indexed field, the second shard entry stores a second flow entry indicator in association with a second flow key determined using the second indexed field, and the second flow entry indicator indicates a second log entry indicator that indicates the entry location.

Aspects described above can be ultimately implemented in a network appliance that includes physical circuits that implement digital data processing, storage, and communications. The network appliance can include processing circuits, ROM, RAM, CAM, and at least one interface (interface(s)). The CPU cores described above are implemented in processing circuits and memory that is integrated into the same integrated circuit (IC) device as ASIC circuits and memory that are used to implement the programmable packet processing pipeline. For example, the CPU cores and ASIC circuits are fabricated on the same semiconductor substrate to form a System-on-Chip (SoC). The network appliance may be embodied as a single IC device (e.g., fabricated on a single substrate) or the network appliance may be embodied as a system that includes multiple IC devices connected by, for example, a printed circuit board (PCB). The interfaces may include network interfaces (e.g., Ethernet interfaces and/or InfiniBand interfaces) and/or PCI Express (PCIe) interfaces. The interfaces may also include other management and control interfaces such as I2C, general purpose IOs, USB, UART, SPI, and eMMC.

As used herein the terms "packet" and "frame" may be used interchangeably to refer to a protocol data unit (PDU) that includes a header portion and a payload portion and that is communicated via a network protocol or protocols. A PDU may be referred to as a "frame" in the context of Layer <NUM> (the data link layer) and as a "packet" in the context of Layer <NUM> (the network layer). For reference, according to the P4 specification: a network packet is a formatted unit of data carried by a packet-switched network; a packet header is formatted data at the beginning of a packet in which a given packet may contain a sequence of packet headers representing different network protocols; a packet payload is packet data that follows the packet headers; a packet-processing system is a data-processing system designed for processing network packets, which, in general, implement control plane and data plane algorithms; and a target is a packet-processing system capable of executing a P4 program.

Instructions or sub-operations of distinct operations may be implemented in an intermittent and/or alternating manner.

It should also be noted that at least some of the operations for the methods described herein may be implemented using software instructions stored on a computer usable storage medium for execution by a computer. As an example, an embodiment of a computer program product includes a computer usable storage medium to store a computer readable program.

Claim 1:
A computer-implemented method comprising:
creating a flow log object (<NUM>; <NUM>; <NUM>) that includes a plurality of log entries (<NUM>, <NUM>, <NUM>, <NUM>) at a plurality of log entry locations, wherein the log entries (<NUM>, <NUM>, <NUM>, <NUM>) are records of a plurality of network traffic flows processed by the network appliances (<NUM>);
creating an index object (<NUM>; <NUM>) that includes a plurality of shards (<NUM>; <NUM>) and a flow table (<NUM>; <NUM>) that indicate the log entry locations; and
storing the flow log object (<NUM>; <NUM>; <NUM>) and the index object (<NUM>; <NUM>) in at least one nonvolatile memory (<NUM>),
wherein
each of the log entries (<NUM>, <NUM>, <NUM>, <NUM>) includes a plurality of indexed field values for a plurality of indexed fields that include a first indexed field and a second indexed field,
the flow table includes a plurality of flow entries (<NUM>, <NUM>) that include a plurality of log entry indicators (<NUM>),
the log entries (<NUM>, <NUM>, <NUM>, <NUM>) include a log entry stored at an entry location,
a first shard entry (<NUM>) is stored in a first one of the shards (<NUM>) indicated by a first shard identifier determined using the first indexed field,
the first indexed field is used to determine a first flow key,
the first shard entry (<NUM>) stores a first flow entry indicator in association with the first flow key,
the first flow entry indicator indicates a first log entry indicator that indicates the entry location,
a second shard entry (<NUM>) is stored in a second one of the shards (<NUM>) indicated by a second shard identifier determined using the second indexed field,
the second indexed field is used to determine a second flow key,
the second shard entry (<NUM>) stores a second flow entry indicator in association with the second flow key, and
the second flow entry indicator indicates a second log entry indicator that indicates the entry location.