axiak/pyre2 — Issue #7: regex slowness
Author: lakiko
Created: 2011-10-11 08:32:23+00:00
State: open

using latest pyre2 and RE2, when trying to proccess a big regex for webserver logs pythons re is far more quick than pyre2, i have added this on performance.py for testing:
weblog structure: '$domain $remote_addr $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'"$request_time" ';

def getweblogdata():
return open('/var/log/apache2/access.log') ##Mine is about 9mb

@register_test("weblog scan",
r'^(([\w\d]|[\w][\w\d\-]*[\w\d])\.)*([\w]|[\w][\w\d\-]*[\w\d])[\s]([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})[\s]*(.+)[\s]\[(.*)\][\s]"(GET|HEAD|POST|PUT|DELETE|OPTIONS)[\s](.*)[\s](HTTP.*)"[\s]([\d]*)[\s]([\d]*)[\s]"(.*)"[\s]"(.+)"[\s]"(.+)"',
1,
data=getweblogdata())
def weblog_matches(pattern, data):
"""
Match weblog data.
"""
total=0
for line in data:
p = pattern.match(line)
if p==None:
continue
total += len(p.groups())
data.seek(0)
return total

|Test |Description |# total runs|``re`` time(s)|``re2`` time(s)|% ``regex`` time|
---------------------------------------------------------------------------------------------------------------------------------------------
|weblog scan |search and extract weblog data. |1 |8.678 |94.259 |1086.13% |

As far as i can see, this probably has to do with the bindings