Abstract:
A system and method for inspecting a data stream for data segments matching one or more patterns each having a predetermined allowable error, which includes filtering a data stream for a plurality of patterns of symbol combinations with a plurality of parallel filter mechanisms, detecting a plurality of potential pattern piece matches, identifying a plurality of potentially matching patterns, reducing the identified plurality of potentially matching patterns to a set of potentially matching patterns with a reduction stage, providing associated data and the reduced set of potentially matching patterns, each having an associated allowable error, to a verification stage, and verifying presence of a pattern match in the data stream from the plurality of patterns of symbol combinations and associated allowable errors with the verification stage.

Description:
FIELD OF THE INVENTION 
     The present invention relates to the field of approximate pattern matching with a large set of patterns. In particular, the present invention relates to a scalable filtering circuit and reduction stage for approximate pattern matching with a large group of patterns. 
     BACKGROUND OF THE INVENTION 
     Approximate pattern or string matching is a significant problem that arises in many important applications. These can include, but are not limited to, computational biology, databases and computer communications. This task includes searching for matches between the specified pattern or set of patterns while typically permitting a specified number of errors. As an example, one may desire to search for the word “queuing” while allowing for two errors. This could return results such as the word “queueing” with one character insertion and “cueing” with one character substitution and one character deletion. By allowing a specified number of errors, this allows the search to catch typical spelling variations or errors and still find the desired pattern. Approximate pattern matching is not only a complex task but requires a tremendous amount of computer resources. 
     Typically, there is a fast filtering step that is followed by the verification step that performs the full approximate matching function. An example of this prior art filtering technique is shown by referring to  FIG. 1  and is generally indicated by numeral  10 . This typical approach is to slice a pattern “P”, as indicated by numeral  12 , into k+1 pattern pieces, which are a sequence of non-overlapping sub-patterns, and search for exact matches between the text and the pattern pieces. In this case, “k” is equal to the number of allowable errors, which is the maximum edit distance ed(T i . . . j ,P), which is indicated in this nonlimiting example by the numeral two (2) as indicated by numeral  14 . 
     A data string T i . . . j    16  is then analyzed for an occurrence of at least one substring of the data string  16  that matches at least one of the non-overlapping sub-patterns associated with pattern “P”  12 . This approach relies on the following properties:
         a. If string S=T a . . . b  matches pattern P with at most k errors, and P=p 1  . . . p j  (a sequence of non-overlapping sub-patterns), then some sub-string of S matches at least one of the p i &#39;s with at most └k/j┘ errors   b. If there are character positions i≦j such that ed(T i . . . j ,P)≦k, then T j−m+1 . . . j  includes at least m-k characters of P where m is the size of the pattern (in characters)   c. Therefore, if we slice P into k+1 pieces (non-overlapping sub-patterns), then at least one of the pieces must match exactly       

     Therefore, if we slice “P”  12  by the total number of errors “k”  14  plus one (1) into non-overlapping sub-pattern pieces then at least one of the non-overlapping sub-pattern pieces must match exactly. As shown in the Example of  FIG. 1 , the data string T i . . . j    16  is divided into k+1 or three (3) pieces of non-overlapping sub-patterns. Therefore the three (3) pieces are “abra” indicated by numeral  18 , “cada” indicated by numeral  20 , and “bra” indicated by numeral  22 . In this example, “cada” indicated by numeral  20  is an exact match with two errors where the letters “br” are replaced and the letter “b” is deleted. 
     There is a significant need for a fast and cost effective mechanism for pattern matching utilizing a substantial amount of input data with a considerable set of potentially matching patterns. 
     SUMMARY OF INVENTION 
     In one aspect of this invention, a method for inspecting a data stream for data segments matching one or more patterns each having a predetermined allowable error with at least one search engine is disclosed. This method includes filtering a data stream for a plurality of patterns of symbol combinations with a plurality of parallel filter mechanisms each configured to detect one or more patterns each with an associated allowable error, detecting a plurality of potential pattern piece matches with the plurality of parallel filter mechanisms, identifying a plurality of potentially matching patterns, each having an associated allowable error, from the plurality of parallel filter mechanisms, reducing the identified plurality of potentially matching patterns to a set of potentially matching patterns, each having an associated allowable error with a reduction stage, providing associated data and the reduced set of potentially matching patterns, each having an associated allowable error, to a verification stage, and verifying presence of a pattern match in the data stream from the plurality of patterns of symbol combinations and associated allowable errors with the verification stage that includes an approximate match engine utilizing the associated data and the reduced set of potentially matching patterns. 
     In another aspect of this invention, a method for inspecting a data stream for data segments matching one or more patterns each having a predetermined allowable error with at least one search engine is disclosed. This method includes filtering a data stream for a plurality of patterns of symbol combinations with a plurality of parallel filter mechanisms each configured to detect one or more patterns each with an associated allowable error, wherein the plurality of parallel filter mechanisms is a group consisting of a set of parallel Bloom filters, a set of parallel Bloom filter arrays or a set of parallel Bloom filter arrays that utilize a single hash key generator, detecting a plurality of potential pattern piece matches with the plurality of parallel filter mechanisms, identifying a plurality of potentially matching patterns, each having an associated allowable error, from the plurality of parallel filter mechanisms, reducing the identified plurality of potentially matching patterns to a set of potentially matching patterns, each having an associated allowable error, providing associated data and the reduced set of potentially matching patterns, each having an associated allowable error, to a verification stage, and verifying presence of a pattern match in the data stream from the plurality of patterns of symbol combinations and associated allowable errors with the verification stage that includes an approximate match engine utilizing the associated data and the reduced set of potentially matching patterns. 
     In still another aspect of this invention, a method and system for inspecting a data stream for data segments matching one or more patterns each having a predetermined allowable error with at least one search engine is disclosed. This method includes utilizing a single hash key generator for extracting a plurality of hash values from a single hash value for inspecting the data stream for data segments matching one or more pattern pieces with false positive errors with at least one search engine, and utilizing the plurality of hash values with a plurality of parallel Bloom filter arrays. 
     In yet another aspect of this invention, a system for inspecting a data stream for data segments matching one or more patterns each having a predetermined allowable error with at least one search engine is disclosed. This system includes a filter stage, which utilizes a plurality of parallel filter mechanisms each configured to detect one or more patterns, each with an associated allowable error, that filter a data stream for a plurality of patterns of symbol combinations and detect a plurality of potential pattern piece matches, and identify a plurality of potentially matching patterns, each having an associated allowable error, a reduction stage, which reduces the identified plurality of potentially matching patterns to a set of potentially matching patterns, each having an associated allowable error, and a verification stage, which includes an approximate match engine, that receives and utilizes associated data and the reduced set of potentially matching patterns and associated allowable errors to verify a presence of a pattern match in the data stream from the plurality of patterns of symbol combinations. 
     In yet another aspect of this invention, a system for inspecting a data stream for data segments matching one or more patterns each having a predetermined allowable error with at least one search engine is disclosed. The system includes a plurality of parallel filter mechanisms, wherein the plurality of parallel filter mechanisms is a group consisting of a set of parallel Bloom filters, a set of parallel Bloom filter arrays or a set of parallel Bloom filter arrays that utilize a single hash key generator, each configured to detect one or more patterns, each with an associated allowable error, that filter a data stream for a plurality of patterns of symbol combinations and detect a plurality of potential pattern piece matches and identify a plurality of potentially matching patterns, each having an associated allowable error, a reduction stage that reduces the identified plurality of potentially matching patterns to a set of potentially matching patterns, each having an associated allowable error, and a verification stage, which includes an approximate match engine, that receives and utilizes associated data and the reduced set of potentially matching patterns and associated allowable errors to verify a presence of a pattern match in the data stream from the plurality of patterns of symbol combinations. 
     Illustrative, but nonlimiting, examples of potential application of the present invention include: an intrusion detection system (IDS) for computer communication networks; computational biology and genetics; text searches for structured and unstructured text; and text searches from optical character scans (OCS). 
     Additional aspects of the present invention include, but are not limited to: a filtering technique for approximate matching with multiple patterns where each pattern may specify its allowable errors that can include a large number of pattern pieces, e.g., tens of thousands of patterns or more; utilizing a parallel set of exact match engines, one for each pattern piece length, to perform parallel match operations and to support a wide variety of (pattern length, allowable error) combinations; allowing each pattern to have a specified number of errors; amenability to parallel hardware search implementation and such implementation can provide fast search results; simplifying a verification stage by limiting the number of potentially matching patterns for a region of text, allowing the verification engine to process additional potential search results in a shorter period of time, which allows the total system to scale in capacity while operating at very high speeds; and utilizing a Bloom filter array for each exact match engine; and efficiently implementing each Bloom filter array by using only one hash function generator. 
     Still another aspect of present invention is the reduction stage wherein the scope of the search in the verification stage is reduced with a smaller set of possibly matching patterns. These techniques use a layer of indirection between pieces and patterns which allows each pattern and its allowable errors to be stored only once. There is a first illustrative technique that simplifies the data structures, making them amenable to hardware implementation. This technique includes a lookup using a bin index to retrieve the piece identifiers for the potentially matching pieces. A second lookup uses the piece identifiers to retrieve the pattern identifiers for the patterns that include the pieces. A third lookup uses the pattern identifiers to retrieve the pattern and associated allowable error pairs to be considered by the verification engine. There is a second illustrative, but nonlimiting technique that utilizes the text pieces that produced matches in the exact match engines to resolve the pattern identifiers for the patterns that include the piece. The pattern identifiers are used to retrieve the pairs of patterns and associated allowable errors to be considered by the verification engine. 
     These are merely some of the innumerable aspects of the present invention and should not be deemed an all-inclusive listing of the innumerable aspects associated with the present invention. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
       For a better understanding of the present invention, reference may be made to the accompanying drawings in which: 
         FIG. 1  provides an illustrative overview of a prior art approximate pattern matching technique; 
         FIG. 2  provides an exemplary block diagram of the present invention including a data source, a filter circuit, a reduction stage and a verification stage; 
         FIG. 3  is an illustrative, but nonlimiting, block diagram of the present invention including Bloom filters with a match detection function; a reduction stage and a verification stage having approximate pattern matching; 
         FIG. 4  provides an exemplary block diagram of a first filtering stage processing technique using a Bloom filter array; 
         FIG. 5  provides an exemplary block diagram of a second filtering stage processing technique using a Bloom filter array with a single hash function generator; 
         FIG. 6  provides an exemplary block diagram of a first reduction stage processing technique; and 
         FIG. 7  provides an exemplary block diagram of a second reduction stage processing technique. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as to obscure the present invention. 
     The present invention is a scalable filtering circuit for approximate pattern matching with a large set of patterns. The filtering circuit checks for potential matches between a set of stored patterns, where each pattern specifies the number of allowable errors, and an input stream of characters. The number of allowable errors is predetermined or specified using the general edit distance measure that counts the number of single character additions, deletions, and substitutions. When a potential match is detected, the location or locations in the input data stream is identified as well as the matching pattern or plurality of potentially matching patterns. The present invention is designed to operate in concert with a verification stage that looks for an approximate match in the data segment(s) of the input data stream utilizing the previously identified potentially matching pattern(s) from the total number of potentially matching patterns. 
     The methodology for searching for a single pattern can be stated as follows: identifying instances of pattern “P” in text “T” with “k” allowable errors, where “k” is the maximum edit distance. The edit distance is defined as the number of single character insertions, deletions, and substitutions with all errors typically, but not necessarily share the same weighting. In general, other types of errors such as transpositions may be included in the distance measure and each type of error may be assigned a unique weight. 
     Assuming that “m” is the size of the pattern in characters and “n” is the size of the text in characters, then the error level can be defined for a particular pattern as the ratio of the number of allowable errors and the size of the pattern, which is “α=k/m”. The error level and the size of the alphabet from which the pattern and text are constructed, “σ,” affect the probability that matches will be found. An expression for the match probability, “f(m,k)” assuming a randomly constructed text and a randomly constructed pattern is provided by the following equation: 
               f   ⁡     (     m   ,   k     )       =       (       ⅇ   2         σ   ⁡     (     1   -   α     )       2       )       m   ⁡     (     1   -   α     )               
This is where “e” is the base of the natural logarithm.
 
     It is believed that filtering algorithms achieve better performance for a variety of approximate pattern matching problems. The general approach being to perform a simple search on a small section of text to identify potential matches. When a potential match is found, the region of text is examined to see if it is, in fact, an approximate match for a specified pattern. In general, verification is performed by any approximate pattern matching algorithm and may be tightly or loosely coupled to a filtering operation. 
     A general schematic of the system of the present invention is shown in  FIG. 2  and is indicated by numeral  30 . Large amounts of data can be provided as input through either a communication link, a disk, a redundant array of independent disks (RAID), or storage area network (SAN), as well as a wide variety of other data sources capable of feeding a filter circuit with high data speed. This data input is indicated by numeral  32  can also be provided through a network input that is indicated by numeral  34 . High speed data can be provided through a high-bandwidth interconnect indicated by numeral  36  at high speeds. This high speed data is then passed through a filter circuit  38  that scans the input data for potential matches for a set of input patterns. There is then a reduction stage  40  between the filter circuit  38  and a verification stage  42  that narrows the set of potentially matching patterns that must be considered by the verification stage  42  when processing data segments that produce a match in the filter circuit  38 . The verification stage  42  performs a full approximate operation to verify whether or not there is a match for a set of input patterns  46 . Search results are then provided as indicated by numeral  44 . Predetermined input patterns  46  are provided to the filter circuit  38 , the reduction stage  40  and the verification stage  42 . 
     In a particular window of text, it is possible to search for an exact match and any of the pattern pieces specified by a predetermined number “r” patterns. If a specific pattern “i” allows “k i ” errors, then the total number of pattern pieces is indicated by the equation: 
     
       
         
           
             p 
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 r 
               
               ⁢ 
               
                 ( 
                 
                   
                     k 
                     i 
                   
                   + 
                   1 
                 
                 ) 
               
             
           
         
       
     
     This filtering approach is utilized with parallel filter mechanisms which can include a parallel set of Bloom filters, a set of parallel Bloom filter arrays, or a set of Bloom filter arrays that utilize a single hash function generator. As shown in  FIG. 3 , which is the schematic of the basic hardware implementation of the present invention is indicated by numeral  50  and includes a number of Bloom filters indicated by numeral  54 . In a classical Bloom filter  54 , elements are inserted into a set using “b” hash values were the element is utilized as the key and where each hash value identifies a bit position in a B-bit vector. The bits at each of the b bit positions are preferably set to one (1). If a bit is already set to one (1), then no change will be made. In order to test whether or not a particular element is a member of the set represented by a Bloom filter  54 , the element and the same b hash functions are utilized to compute b hash values. If all the b bits in the vector are set to one (1), then the element is declared to be a member of the set. 
     A Bloom filter  54  will not produce a false negative. If an element is a member of a set, then the b bit positions in the B-bit vector are set to one (1) when the element is inserted into the set. The insertion of additional elements in the set does not reset any of the bits in the vector. However, Bloom filters  54  do produce false positives with a determined probability. This probability can be computed by the equation: 
             f   =       (     1   -     ⅇ       -   pb     B         )     b           
If the following relationship holds:
 
             b   =       B   p     ⁢   ln   ⁢           ⁢   2           
then: ƒ=(½) b   
     Approximate match filtering on multiple patterns and allowing each pattern to specify its allowable errors produces sets of pattern pieces of various lengths. Preferably, but not necessarily, the bloom filters  54  store fixed-length elements with one bloom filter circuit  54  for each possible pattern piece length. Therefore, the range of possible pattern piece lengths are constrained within a range. 
     If l min  is the minimum pattern piece length then l min  is less than or equal to the value of the size of the pattern m divided by the maximum edit distance k plus one (1): TK 2 Z 
     
       
         
           
             
               l 
               min 
             
             ≤ 
             
               ⌊ 
               
                 m 
                 
                   k 
                   + 
                   1 
                 
               
               ⌋ 
             
           
         
       
     
     If l max  is the maximum piece length, then l max  is greater than or equal to the value of the size of the pattern m divided by the maximum edit distance k plus one (1): 
     
       
         
           
             
               ⌈ 
               
                 m 
                 
                   k 
                   + 
                   1 
                 
               
               ⌉ 
             
             ≤ 
             
               l 
               max 
             
           
         
       
     
     The total number of Bloom filters  54  that are required when each Bloom filter of the Bloom filters  54  corresponds to a pattern piece length is:
 
 l   max   −l   min +one   (1).
 
     A preferred approach is to query each of the Bloom filters  54  in parallel, as shown by the schematic provided by numeral  50  in  FIG. 3 . Each one of the Bloom filters  54  correspond to a pattern piece length so that various strides of the text window can be selected as an input key to each Bloom filter  54 . 
     If any of the Bloom filters  54  result in a detected match  56 , then the segment of data or text window  58 , the location of the segment of data in the input stream  59  and additional match meta data  60  are sent to a reduction stage  40 . The techniques utilized in the reduction stage  40  can narrow the potentially matching patterns significantly, e.g., over 10,000 to less than 10. 
     The result passing from the reduction stage  40  goes to the verification stage  42 , which includes the approximate match search engine. By reducing the number of candidate patterns to be considered by the verification stage  42 , allowing the verification stage to process more potential search results in a given amount of time, and thus allowing the total system to scale in capacity while operating at high speeds. 
     A Bloom filter array  54  typically minimizes the number of memory accesses, e.g., random access memory (RAM), required for a set membership query. Moreover, the Bloom filter array  54  partitions the B-bit vector into “W” vectors of size “q=B/W” where “q” is the word size of the memory. There can preferably be an even distribution of stored elements over the “W” vectors (memory words) using a pre-filter hash function. This creates an array of “W” “q”-bit Bloom filters. 
     The bits in the “q”-bit Bloom filters  54  are set, during programming, and then checked, during queries, using “b” hash functions. Querying a Bloom filter array  54  requires one (1) memory read to fetch the “q”-bit vector. Using a register  80  and bit-select circuitry  82 , checking the bit locations specified by “b” hash functions may be performed on-chip, in a pipelined fashion, as shown in  FIG. 4  and generally indicated by numeral  70 . 
     In this application, a key (i.e. pattern piece) is indicated by numeral  72 . The key is used by the pre-filter hash function  73  to identify a particular “q”-bit vector in the listing of “w” vectors. The particular “q”-bit vector in the listing of “w” vectors is indicated by column  74  in memory, e.g., RAM, wherein a particular and illustrative vector is identified by numeral  76 . These queries are checked with series of “b” hash functions indicated by numeral  78  identifying bit positions within a register  80 , which is then provided to a match detection function  82 . If all “b” bit positions are set to a one (1), then the key is a pattern piece for a potentially matching pattern. 
     Preferably, the amount of logic required to implement a Bloom filter array  54  can be minimized, as shown in  FIG. 5 . This logic is generally indicated by numeral  90 . There is a single hash function indicated by numeral  92 . There is the generation of a single random value. A subset of the bits from this random value are utilized to construct the pre-filter hash address and “b” filter bit-positions that is indicated by numeral  94 . The particular “q”-bit vector in the listing of “w” vectors is indicated by column  96  in memory, e.g., RAM, wherein a particular and illustrative vector is identified by numeral  98 . This particular vector  98  is passed to a register  100  and then on to a match pattern detection function  102 . This Bit select value  94  must be at least log 2 (W)+(b*log 2 (q)) bits in size. The H 3  class of hash functions  92  is an illustrative, but nonlimiting, example of hash functions that can produce wide enough values for this application. 
     An illustrated, but nonlimiting example of reconfigurable hardware that could be utilized includes FPGAs, i.e., field programmable gate arrays, which includes a Xilinx® VirtexII® 4000 series FPGA. Xilinx, Inc., is a Delaware corporation, having a place of business at 2100 Logic Drive, San Jose, Calif. 95124-3400. An illustrated, but nonlimiting, example of the embedded memory in the VirtexII® series of devices include One Hundred and Twenty (120) of the eighteen (18) kilobyte block random access memories (BlockRAMs). These BlockRAMs can be configured to various size words with an illustrative, but nonlimiting, maximum word length of thirty-six (36) bits by Five Hundred and Twelve (512) words. 
     Utilizing the previous expression for probability of a false positive and assuming uniform hashing performance, the Bloom filter array  54  implemented with an eighteen (18) kilobyte BlockRAM can represent a set of 3,194 elements with a false probability of 0.063 when the number of b bit positions equals four (4). When number of b bit positions equals three (3), the capacity increases to 4,259 elements but the false positive probability increases to 0.125. 
     As previously stated, one Bloom filter array  54  is required for each unique pattern piece length. There is also consideration of the number of parallel circuits that are required to keep pace with the data input rate. In an illustrative, but nonlimiting example, a system that accepts eight (8) new ASCII characters per cycle (64-bit interface) requires eight (8) instances of the circuit operating in parallel. For the VirtexII 4000® FPGA, there are at most fifteen (15) BlockRAMs available for each circuit instance. In order to have BlockRAM resources available for interface buffers, this results in limiting the BlockRAMs to a lower number, e.g., fourteen BlockRAMs. This illustrative, but nonlimiting, resource allocation can place a constraint on the length of pattern pieces and the combination of pattern piece and allowable error. This in turn places a limit on the maximum error level. When “m” is equal to the size of the pattern, “p” is equal to number of pattern pieces and “k” is the number of allowable errors or edit distance then the following equations are applicable:
 
 m   min   =p   min ( k+ 1)
 
 m   max   =p   max ( k+ 1)
 
               α   max     =       k     m   min       =     k       p   min     ⁡     (     k   +   1     )                 
where α=ratio of the number of errors divided by the size of the pattern.
 
     The following Table 1 is when “α”, i.e., ratio of the number of errors divided by the size of the pattern is less than or equal to one (1): 
     
       
         
               
               
               
             
           
               
                 TABLE 1 
               
               
                   
               
               
                 k 
                 m min   
                 m max   
               
               
                   
               
             
             
               
                 0 
                 1 
                 14 
               
               
                 1 
                 2 
                 28 
               
               
                 2 
                 3 
                 42 
               
               
                 3 
                 4 
                 56 
               
               
                 . . . 
                 . . . 
                 . . . 
               
               
                   
               
             
          
         
       
     
     The following Table 2 is when “α”, i.e., ratio of the number of errors divided by the size of the pattern is less than or equal to one-half (½): 
     
       
         
               
               
               
             
           
               
                 TABLE 2 
               
               
                   
               
               
                 k 
                 m min   
                 m max   
               
               
                   
               
             
             
               
                 0 
                 2 
                 15 
               
               
                 1 
                 4 
                 30 
               
               
                 2 
                 6 
                 45 
               
               
                 3 
                 8 
                 60 
               
               
                 . . . 
                 . . . 
                 . . . 
               
               
                   
               
             
          
         
       
     
     The following Table 3 is when “α”, i.e., ratio of the number of errors divided by the size of the pattern is less than or equal to one-third (⅓): 
     
       
         
               
               
               
             
               
               
               
             
           
               
                 TABLE 3 
               
               
                   
               
               
                 k 
                 m min   
                 m max   
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 0 
                 3 
                 16 
               
               
                 1 
                 6 
                 32 
               
               
                 2 
                 9 
                 48 
               
               
                 3 
                 12 
                 64 
               
               
                 . . . 
                 . . . 
                 . . . 
               
               
                   
               
             
          
         
       
     
     Therefore in the second example and Table 2, when “α”, i.e., ratio of the number of errors divided by the size of the pattern is less than or equal to one-half (½) and when the errors, i.e., “k=0”, there must be at least two (2) characters and no more than fifteen (15) characters. A pattern that allows one error, i.e., “k=1”, must contain at least four (4) characters and no more than thirty (30) characters. Although a wide variety of admissible pattern sizes and allowable errors can be utilized, it is believed that a pattern is at least two (2) characters and no more than fifteen (15) characters will be a workable constraint for most text searches in English, however, this should not be construed as a limit. 
     A rough capacity estimate can be developed by assuming that pieces are uniformly distributed over a range of allowable lengths. In an illustrative, but nonlimiting example, if each Bloom filter array  54  has a capacity of approximately Three Thousand (3,000) pattern pieces then the system has an aggregate capacity of Forty-Two Thousand (42,000) pattern pieces. If it is assumed that each pattern can be divided into three (3) pattern pieces, then the system has a capacity of Fourteen Thousand (14,000) patterns. 
     Once a potential match has been detected one or more pattern piece lengths  56 , as shown in  FIG. 3 , then the region of text must be examined by the verification stage  42  in order to determine whether or not there is an approximate match for one of the “r” patterns. Since “r” may be on the order of 10,000 patterns or more, there is a need to narrow the scope of the search that the verification stage  42  must perform. This is where the reduction stage  40  provides a valuable role of reducing the set of possible matching patterns (pattern set) for the verification stage  42  to consider. As long as the parameters fall within the constraints, there is an assumption that the number of allowable errors may be specified for each pattern. 
     Throughout this patent application as shown in  FIG. 3 , the filter stage  38 , the reduction stage  40  and/or the verification stage  42 , as shown in  FIG. 3 , can use at least one reconfigurable logic device, e.g., Field Programmable Gate Array (“FPGA”) or at least one integrated circuit, e.g., Application-Specific Integrated Circuit (“ASIC”). 
     There are two illustrative, but nonlimiting, approaches to perform the reduction stage  40 . The first approach is to simplify data searches utilized to resolve the pattern set indicated by numeral  120  in  FIG. 6 . This allows a data string to come into a shift register  52 , which then passes into a filter stage or circuit  38  that preferably includes a Bloom filter array  54 . With this approach, the objective is to utilize some, if not all, of the hash values computed by the Bloom filter array  54  in the filter stage or circuit  38  as an index into a table  128 , e.g., BinIndex  124 . 
     For example a pattern piece  121  can be received by the filter stage or circuit  38  and is received as hash values  126  comprising the BinIndex  124 . The entries in this first table  128  contain identifiers  127  for the pattern pieces, e.g., PieceIDs, which map to the hash values  126  comprising the BinIndex  124 . For example, there is an illustrative identifier, e.g., PieceID 1  and PieceID 4 , which is associated with the example pattern piece  121 . The identifiers  127  for the pattern pieces, e.g., PieceIDs, are a unique binary tag assigned to each pattern piece. 
     There is a second table  132  that utilizes the identifiers  128  for the pattern pieces, e.g., PieceIDs, to index one or more pattern identifiers for the set of potentially matching patterns. Since one or more patterns may specify a particular pattern piece, the entries in the second table  132  contain one or more pattern identifiers, e.g., PIDs. Pattern identifiers, e.g., PIDs, are unique binary tags associated with each pattern. For example, with the illustrative identifier,  131 , relates to two patterns  137  and  138 . 
     There is a third table  134  that utilizes the pattern identifiers, e.g. PIDs, to index one or more (pattern, allowable error) pairs. The identified set or plurality of potentially matching patterns, each with predetermined allowable errors, e.g. (pattern, allowable error) pairs  137  and  138 , is then created as indicated by numeral  136 . 
     This pattern set of potential matches  136  is passed onto the verification stage  42 , which includes evaluation by an approximate match engine  142  for matching patterns and associated predetermined allowable errors. There only needs to be one copy of tables for the pattern piece identifiers  128 , e.g., PieceIDs, pattern identifiers  132 , e.g., PIDs and patterns  134  so long as the number of lookups per cycle does not exceed the amount of lookups supported by the memories. 
     There is a second and preferred methodology for the reduction stage  40 , which is generally indicated by numeral  150  in  FIG. 7 . This allows a data string to pass into a filter stage or circuit  38  that preferably includes a Bloom filter array  54 . With this approach, actual data segments or pieces that produce a match in the Bloom filter array  54  are utilized to resolve the pattern identifiers for the patterns  162  that specify those pattern pieces. The data segments or pieces that produce a match are used to identify one or more entries in data structures, indicated by numerals  156 ,  158  and  160 , which store pattern identifiers, e.g., PIDs, for the patterns that specify the associated pattern pieces. Illustrative, but nonlimiting examples of suitable data structures are a hash table and a balanced search tree. The methodology may include one or more data structures. In the illustrative, but nonlimiting example  150  in  FIG. 7 , one data structure is allocated for each pattern piece length. 
     In an illustrative, but nonlimiting example, two data segments or pieces that produce a match in the Bloom filter array  54  are identified by numerals  121  and  123 . Data piece  121  identifies the entry  157  in the data structure as part of the reduction stage  40 . These data structures  156 ,  158  and  160  can include a wide variety of different structures, e.g., decision tree, hash table, and so forth. The result of these lookup(s) is a set of pattern identifiers for the patterns in the pattern set. As with the prior approach, these pattern identifiers, e.g., PIDs,  156 ,  158  and  160  are utilized to retrieve patterns and associated allowable errors from a table  40  prior to a step of verification. The previously referenced entry  157  identify patterns  163  and  165  and associated allowable errors to produce a set of potential matches  164 . The set of potential matches and associated predetermined allowable errors  164  is then evaluated by an approximate match engine  166  in the verification stage  42 . 
     This approach resolves the set of pattern identifiers in a single step instead of two steps and also eliminates false positive errors produced by the Bloom filter arrays  54 . Also, since the actual data segment is utilized to locate entries in the pattern identifier structure(s), an explicit match is performed. If there is no entry in the table, then a false positive is detected and no pattern identifiers, e.g., PIDs, are passed onto the verification stage  42  for that particular pattern piece length. The tradeoff is that the data structures can be more complex and the implementation more resource intensive depending on the implementation. 
     Therefore, this is a scalable design for a filtering circuit  38  and reduction stage  40  for approximate pattern matching on multiple patterns that are amenable to hardware implementation. In addition to the thousands of patterns, multiple filter circuits can support multiple input symbols per cycle. Utilizing the high performance filtering circuit  38  and the reduction stage  40 , the performance requirements placed on a verification stage  40  can be analyzed. For the purpose of this analysis, we assume that all patterns specify the same number of allowable errors, “k”. Effective load on the verification stage is determined as the probability of a match and the expected size of the set of potentially matching patterns. The probability of match is simply the sum of the probability that any of the “r(k+1)” pieces produce a match in a text window and the false positive probability of the Bloom filter arrays  54  where “r” is the predetermined number of “r” pattern pieces and “k” is the number of predetermined errors. 
     If “L” is the number of Bloom filter arrays  54  and assuming random text on an alphabet of “σ” characters, the probability of any of these pieces matching would be: 
     
       
         
           
             
               E 
               ⁡ 
               
                 [ 
                 match 
                 ] 
               
             
             = 
             
               
                 r 
                 ⁡ 
                 
                   ( 
                   
                     k 
                     + 
                     1 
                   
                   ) 
                 
               
               
                 σ 
                 
                   ⌊ 
                   
                     m 
                     
                       k 
                       + 
                       1 
                     
                   
                   ⌋ 
                 
               
             
           
         
       
     
     The addition of the false positive probability of the Bloom filter arrays provides: 
     
       
         
           
             
               E 
               ⁡ 
               
                 [ 
                 match 
                 ] 
               
             
             = 
             
               
                 
                   r 
                   ⁡ 
                   
                     ( 
                     
                       k 
                       + 
                       1 
                     
                     ) 
                   
                 
                 
                   σ 
                   
                     ⌊ 
                     
                       m 
                       
                         k 
                         + 
                         1 
                       
                     
                     ⌋ 
                   
                 
               
               + 
               Lf 
             
           
         
       
     
     By utilizing illustrative, but nonlimiting, example values L=14, σ=40, r=14,000 and f=0.0034, the match probability is highly sensitive to the minimum piece length. For example, when m=5 and k=1 (a minimum piece length of two (2) characters), then the match probability is one (1). If the pattern size is increased to six (6) (a minimum pattern piece length of three (3) characters), then the match probability is 0.079. This result suggests that the minimum pattern piece size should be three (3) characters or more. In this situation, there will be an expectation of one match every twelve (12) cycles. 
     Utilizing the reduction stage  40  methodology shown in  FIG. 6  with a series of index lookups, then given that at least one Bloom filter array  54  produces a match, the expected number of Bloom filter arrays  54  that will produce a match, i.e., the expected number of bin index lookups, is: 
     
       
         
           
             
               E 
               ⁡ 
               
                 [ 
                 bins 
                 ] 
               
             
             ≤ 
             
               1 
               + 
               
                 L 
                 
                   σ 
                   
                     ⌊ 
                     
                       m 
                       
                         k 
                         + 
                         1 
                       
                     
                     ⌋ 
                   
                 
               
               + 
               Lf 
             
           
         
       
     
     Under the assumption that the pattern pieces are uniformly distributed over “L” Bloom filter arrays  54  (pattern piece lengths) and uniformly distributed over the bins, the expected number of pattern piece identifiers  128 , as shown in  FIG. 6 , per bin is: 
               E   ⁡     [     PieceIds   ⁢     /     ⁢   bin     ]       ≤       r   ⁡     (     k   +   1     )           LW   ⁡     (     B   W     )       b             
where “B” is the size of memory used to implement the Bloom filter array  54 , “W” is the number of words in the Bloom filter array  54  and “b” is the number of hash functions used in each Bloom filter in the Bloom filter array  54 .
 
     Finally, the expected number of patterns that specify a given pattern piece  131 , as shown in  FIG. 6 , include: 
     
       
         
           
             
               E 
               ⁡ 
               
                 [ 
                 
                   patterns 
                   ⁢ 
                   
                     / 
                   
                   ⁢ 
                   PieceIDs 
                 
                 ] 
               
             
             ≤ 
             
               1 
               + 
               
                 r 
                 
                   σ 
                   
                     ⌊ 
                     
                       m 
                       
                         k 
                         + 
                         1 
                       
                     
                     ⌋ 
                   
                 
               
             
           
         
       
     
     Therefore, provided that at least one Bloom filter array  54  produces a match, the expected pattern size is: 
     
       
         
           
             
               E 
               ⁡ 
               
                 [ 
                 patterns 
                 ] 
               
             
             ≤ 
             
               
                 ( 
                 
                   1 
                   + 
                   
                     L 
                     
                       σ 
                       
                         ⌊ 
                         
                           
                             m 
                             / 
                             k 
                           
                           + 
                           1 
                         
                         ⌋ 
                       
                     
                   
                   + 
                   Lf 
                 
                 ) 
               
               ⁢ 
               
                 ( 
                 
                   
                     r 
                     ⁡ 
                     
                       ( 
                       
                         k 
                         + 
                         1 
                       
                       ) 
                     
                   
                   
                     
                       LW 
                       ⁡ 
                       
                         ( 
                         
                           B 
                           W 
                         
                         ) 
                       
                     
                     b 
                   
                 
                 ) 
               
               ⁢ 
               
                 ( 
                 
                   1 
                   + 
                   
                     r 
                     
                       σ 
                       
                         ⌊ 
                         
                           m 
                           
                             k 
                             + 
                             1 
                           
                         
                         ⌋ 
                       
                     
                   
                 
                 ) 
               
             
           
         
       
     
     Assuming uniform text, uniform distribution of piece lengths, uniform distributions (good hash functions) and with L=14, σ=40, r=14,000, b=3 and W=512, then the expected pattern set size is less than ten (10) for practical values of m and k (pattern length and allowable errors). The expected pattern set size quickly approaches one (1) as the alphabet size and/or pattern size increases. 
     The expression for the expected pattern set size when using other reduction stage approach indicated by  FIG. 7  with the text segment as an index are similar and produce a slightly smaller pattern set size. In combination with the previous result, which is one pattern match every twelve (12) cycles, a conservative constraint for the average throughput of the verification stage  42  is approximately one (1) match on one pattern per every cycle. 
     Thus, there has been shown and described several embodiments of a novel invention. As is evident from the foregoing description, certain aspects of the present invention are not limited by the particular details of the examples illustrated herein, and it is therefore contemplated that other modifications and applications, or equivalents thereof, will occur to those skilled in the art. The terms “have,” “having,” “includes” and “including” and similar terms as used in the foregoing specification are used in the sense of “optional” or “may include” and not as “required.” Many changes, modifications, variations and other uses and applications of the present construction will, however, become apparent to those skilled in the art after considering the specification and the accompanying drawings. All such changes, modifications, variations and other uses and applications which do not depart from the spirit and scope of the invention are deemed to be covered by the invention which is limited only by the claims that follow.