So the headline field is supposed to contain that information, but I see
that for instance the Call of Duty Wiki title has more information than
what's in the Solr document. We can keep this in for now, but please do an
audit to see what we're actually missing compared to the headline field for
the wiki document for the top 500 wikis. If enough wikis are using some
alternate promote data field than what we already have, that should be part
of this heuristic. But I would rather we store this information in Solr
during the normal indexing process than re-scraping this data every time we
need to get this information out.


On Thu, Mar 20, 2014 at 9:40 AM, tristaneuan <notifications@github.com>wrote:

> In wikia_dstk/identify/scraping.py:
>
> > @@ -0,0 +1,46 @@
> > +import requests
> > +import traceback
> > +from bs4 import BeautifulSoup
> > +from urllib2 import urlopen
> > +
> > +
> > +def guess_from_title_tag(wid):
>
> The Solr document doesn't contain the HTML tags, only the raw text. The
> tag is likely to contain the most relevant information regarding the
> subject of the wiki, so the purpose of this function is to isolate that
> info.
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/Wikia/data-science-toolkit/pull/2/files#r10803480>
> .
>