Package: os-urlpattern
os-urlpatternThis package is used for unsupervised URLs clustering. Furthermore, it generate URL patterns(RegEx)
from clusters for matching purpose. It is a pure python package tested under python2.7 python3.6,pypycan also be used for performance(4x-8x). Command line tools are provided
for standalone clustering and matching, APIs are also convenient. Several extra packages can be
installed for additional features. Under CPython 1cpu, 100 thousand URLs clustering cost almost 1min
and 200M memory. Built-in matching strategy is efficient enough in most use cases(4k/s, depend on
patterns complexity).$pipinstall-Uos-urlpattern$wget-qO-'https://git.io/f4QlP'|pattern-make/[0-9]{2}[\.]html
      http://example.com/01.html
      http://example.com/02.html
      http://example.com/03.html
/[0-9]{3}/test[0-9]{2}[\.]html
      http://example.com/123/test01.html
      http://example.com/456/test02.html
      http://example.com/789/test03.htmlAknowledgementSimilar URLsURLs with the sameURL structure.Components of the parsed URLs at the same position are in the samecharacter space.Different types of charactors may be in the same order in most cases.URL structureTypically, URL can be parsed into 6 components:<scheme>://<netloc>/<path>;<params>?<query>#<fragment>Because different sites may have similar URLs structure and <params> is rare, so <schema>
<netloc> and <params> are ignored, <path> <query> <fragment> are used to define URL structure.If the URLs have the same path levels, same query keys(also keys order) and with the same
fragment existence, their URL structure should be the same.http://example.com/p1/p2?k1=v1&k2=v2#pos

URL structure:
path levels: 2
query keys: k1, k2
have fragment: TrueCharacter spaceConsiderRFC 3986 (Section 2: Characters),
URL with the following characters would be legal:ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;=%<>\"{}^|There are three major character space: lower-case letters(a-z), upper-case letters(A-Z),
number letters(0-9). Other symbols are in their own character space.HeLlOwoRd666!

character space: a-z A-Z 0-9 !Order considerationSplit a string by character, consecutive character space can be joined. In most cases, order is a
distinguished feature.HELLOword666!

split into: HELLO word 666 !

character space order: A-Z a-z 0-9 !MixComplex consecutive major character space can be mixed, order is less important.HellWorld666!

split into: H ell W orld 666 !

major join: HellWorld666 !

character space order: A-Za-z0-9 !Because of URL quote, ‘%’ can be mixed with major character space.%E4%BD%A0%E5%A5%BD!

split into: % E 4 % BD % A 0 % E 5 % A 5 % BD !

major join: %E4%BD%A0%E5%A5%BD !

character space order: A-Z0-9% !URL patternURL pattern is used to express each cluster. It is normal regex string. Each URL in
the same cluster can be matched with the pattern.pattern examples:

/news/[0-9]{8}/[a-z]+[\\.]html
/newsShow[\\.]asp[\\?]dataID=[0-9]+
/thread[\\-][0-9]+[\\-][0-9][\\-]1[\\.]htmlThe built-in matching strategy is strict, it can’t tolerate incomplet matching.letter: helloword

pattern01: [a-z0-9]+  # not match, because no number in the letter
pattern02: [a-z]+ # matchInstallInstall with pip$ pip installos-urlpatternInstall extra packagessubpackageinstall commandenablesmemorypip installos-urlpattern[memroy]Show memory useageete-treepip installos-urlpattern[ete-tree]Enableetepattern tree formatterUsageCommand linepattern-makeLoad urls, cluster and dump patterns.$pattern-make-husage: pattern-make [-h] [-v] [-i INPUTS [INPUTS ...]]
                    [-l {NOTSET,DEBUG,INFO,WARN,ERROR,FATAL}] [-c CONFIG]
                    [-f {PATTERN,CLUSTER,JSON,ETE,INLINE,NULL}]

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -i INPUTS [INPUTS ...], --inputs INPUTS [INPUTS ...]
                        input files to be processed (default: stdin)
  -l {NOTSET,DEBUG,INFO,WARN,ERROR,FATAL}, --loglevel {NOTSET,DEBUG,INFO,WARN,ERROR,FATAL}
                        log level (default: NOTSET)
  -c CONFIG, --config CONFIG
                        config file
  -f {PATTERN,CLUSTER,JSON,ETE,INLINE,NULL}, --formatter {PATTERN,CLUSTER,JSON,ETE,INLINE,NULL}
                        output formatter (default: CLUSTER)Dump clustered URLs with patterns:$caturls.txt|pattern-make-Ldebug>clustered.txtOnly generate URL patterns:$caturls.txt|pattern-make-Ldebug-Fpattern>patterns.txtGenerate pattern tree from URLs(eteinstalled):$caturls.txt|pattern-make-Ldebug-Fetepattern-matchLoad patterns, dump URLs matched results.$pattern-match-husage: pattern-match [-h] [-v] [-i INPUTS [INPUTS ...]]
                     [-l {NOTSET,DEBUG,INFO,WARN,ERROR,FATAL}] -p
                     PATTERN_FILES [PATTERN_FILES ...] [-a]

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -i INPUTS [INPUTS ...], --inputs INPUTS [INPUTS ...]
                        input files to be processed (default: stdin)
  -l {NOTSET,DEBUG,INFO,WARN,ERROR,FATAL}, --loglevel {NOTSET,DEBUG,INFO,WARN,ERROR,FATAL}
                        log level (default: NOTSET)
  -p PATTERN_FILES [PATTERN_FILES ...], --pattern-files PATTERN_FILES [PATTERN_FILES ...]
                        pattern files to be loaded
  -a, --all-matched     all matched patternsMatch URLs:$caturls.txt|pattern-match-Ldebug-ppatterns.txtAPIsCluster and generate URL patterns:fromos_urlpattern.formatterimportpformatfromos_urlpattern.pattern_makerimportPatternMakerpattern_maker=PatternMaker()# load URLs(unicode)forurlinurls:pattern_maker.load(url)# cluster and print patternforurl_meta,clusteredinpattern_maker.make():forpatterninpformat('pattern',url_meta,clustered):# do whatever you wantpassMatch URLs:fromos_urlpattern.pattern_matcherimportPatternMatcherpattern_matcher=PatternMatcher()# load url_pattern(unicode)forurl_patterninurl_patterns:# meta will bind to matched resultpattern_matcher.load(url_pattern,meta=url_pattern)# match URL(unicode)forurlinurls:matched_results=patterm_matcher.match(url)# the best matched result:# sorted(matched_results, reverse=True)[0]patterns=[n.metaforninmatched_results]Low-level APIs:It is necessary to use low-level APIs for customizing processing procdure,
especially for parallel computing or working on an distributed cluster(hadoop).Key points: same fuzzy-digest same maker and same matcher.Useos_urlpattern.parser.fuzzy_digestto get fuzzy digest from URL,
URL pattern or URLMeta and parsed pieces/patterns.A brief All-In-One example:from__future__importprint_function,unicode_literalsfromos_urlpattern.formatterimportpformatfromos_urlpattern.parserimportfuzzy_digest,parsefromos_urlpattern.pattern_makerimportMakerfromos_urlpattern.pattern_matcherimportMatcherurls=['http://t.com/%02d.html'%iforiinxrange(0,10)]makers={}matchers={}# Init makers from URLs(unicode).forurlinurls:url_meta,parsed_pieces=parse(url)# same digest same makerdigest=fuzzy_digest(url_meta,parsed_pieces)ifdigestnotinmakers:makers[digest]=Maker(url_meta)makers[digest].load(parsed_pieces)# Iterate makers, do clustering, generate URL pattern and init matchers.formakerinmakers.values():forclusteredinmaker.make():forpatterninpformat('pattern',maker.url_meta,clustered):# init matchersurl_meta,parsed_patterns=parse(pattern)digest=fuzzy_digest(url_meta,parsed_patterns)ifdigestnotinmatchers:matchers[digest]=Matcher(url_meta)matchers[digest].load(parsed_patterns,pattern)# Match URLs(unicode).forurlinurls:url_meta,parsed_pieces=parse(url)# same digest same matcherdigest=fuzzy_digest(url_meta,parsed_pieces)ifdigestinmatchers:matched=[n.metaforninmatchers[digest].match(parsed_pieces)]print(url,*matched,sep="\t")else:# no matched at allpassUnit Tests$ toxLicenseMIT licensed.