Package: serpextract
serpextractprovides easy extraction of keywords from search engine results pages (SERPs).This module is possible in large part to the very hard work of theMatomoteam.
Specifically, we make extensive use of theirlist of search engines.InstallationLatest release on PyPI:$ pip install serpextractUsageCommand LineCommand-line usage, returns the engine name and keyword components separated by a
comma and enclosed in quotes:$ serpextract "http://www.google.ca/url?sa=t&rct=j&q=ars%20technica"
"Google","ars technica"You can also print out a list of all the SearchEngineParsers currently available in
your local cache via:$ serpextract -lPythonfromserpextractimportget_parser,extract,is_serp,get_all_query_paramsnon_serp_url='http://arstechnica.com/'serp_url=('http://www.google.ca/url?sa=t&rct=j&q=ars%20technica&source=web&cd=1&ved=0CCsQFjAA''&url=http%3A%2F%2Farstechnica.com%2F&ei=pf7RUYvhO4LdyAHf9oGAAw&usg=AFQjCNHA7qjcMXh''j-UX9EqSy26wZNlL9LQ&bvm=bv.48572450,d.aWc')get_all_query_params()# ['key', 'text', 'search_for', 'searchTerm', 'qrs', 'keyword', ...]is_serp(serp_url)# Trueis_serp(non_serp_url)# Falseget_parser(serp_url)# SearchEngineParser(engine_name='Google', keyword_extractor=['q'], link_macro='search?q={k}', charsets=['utf-8'])get_parser(non_serp_url)# Noneextract(serp_url)# ExtractResult(engine_name='Google', keyword=u'ars technica', parser=SearchEngineParser(...))extract(non_serp_url)# NoneNaive DetectionThe list of search engine parsers that Matomo and thereforeserpextractuses is far from
exhaustive.  If you wantserpextractto attempt to guess if a given referring URL is a SERP,
you can specifyuse_naive_method=Truetoserpextract.is_serporserpextract.extract.
By default, the naive method is disabled.Naive search engine detection tries to find an instance ofr'\.?search\.'in thenetlocof a URL.  If found,serpextractwill then try to find a keyword in thequeryportion of
the URL by looking for the following params in order:_naive_params = ('q', 'query', 'k', 'keyword', 'term',)If one of these are found, a keyword is extracted and anExtractResultis constructed as:ExtractResult(domain, keyword, None)  # No parser, but engine name and keyword# Not a recognized search engine by serpextractserp_url='http://search.piccshare.com/search.php?cat=web&channel=main&hl=en&q=test'is_serp(serp_url)# Falseextract(serp_url)# Noneis_serp(serp_url,use_naive_method=True)# Trueextract(serp_url,use_naive_method=True)# ExtractResult(engine_name=u'piccshare', keyword=u'test', parser=None)Custom ParsersIn the event that you have a custom search engine that you’d like to track which is not currently
supported by Matomo/serpextract, you can create your own instance ofserpextract.SearchEngineParserand either pass it explicitly to eitherserpextract.is_serporserpextract.extractor add it
to the internal list of parsers.# Create a parser for PiccSharefromserpextractimportSearchEngineParser,is_serp,extractmy_parser=SearchEngineParser(u'PiccShare',# Engine nameu'q',# Keyword extractoru'/search.php?q={k}',# Link macrou'utf-8')# Charsetserp_url='http://search.piccshare.com/search.php?cat=web&channel=main&hl=en&q=test'is_serp(serp_url)# Falseextract(serp_url)# Noneis_serp(serp_url,parser=my_parser)# Trueextract(serp_url,parser=my_parser)# ExtractResult(engine_name=u'PiccShare', keyword=u'test', parser=SearchEngineParser(engine_name=u'PiccShare', keyword_extractor=[u'q'], link_macro=u'/search.php?q={k}', charsets=[u'utf-8']))You can also permanently add a custom parser to the internal list of parsers thatserpextractmaintains so that you no longer have to explicitly pass a parser
object toserpextract.is_serporserpextract.extract.fromserpextractimportSearchEngineParser,add_custom_parser,is_serp,extractmy_parser=SearchEngineParser(u'PiccShare',# Engine nameu'q',# Keyword extractoru'/search.php?q={k}',# Link macrou'utf-8')# Charsetadd_custom_parser(u'search.piccshare.com',my_parser)serp_url='http://search.piccshare.com/search.php?cat=web&channel=main&hl=en&q=test'is_serp(serp_url)# Trueextract(serp_url)# ExtractResult(engine_name=u'PiccShare', keyword=u'test', parser=SearchEngineParser(engine_name=u'PiccShare', keyword_extractor=[u'q'], link_macro=u'/search.php?q={k}', charsets=[u'utf-8']))TestsThere are some basic tests for popular search engines, but more are required:$ pip install -r requirements.txt
$ py.testCachingInternally, this module caches an OrderedDict representation ofMatomo’s list of search engineswhich is stored inserpextract/search_engines.json.  This isn’t intended to change that often and so this
module ships with a cached version.