fix gb install 1-15 logic
fix delete collection api
started adding redownload logic.
just print out cached page straight, it is
just the diffbot json reply pretty much
verbatim, except for being tokenized.
should no longer escape forward slashes.
take out confusing function no longer used
always use kstart.
fixed restrictDomain bug of not saving parm.
sped up csv download around 2x.
fixed core
ignore empty products (i.e. {}) when tokenizing diffbot reply
take out potentially bad robots.txt
filter compression logic.
fixed form input. fixed page parser submission.
added ability to dump out termlist from posdb
like type:json (with a colon in it) to try to debug
msft seeing html in csv output.
fix a couple cores related to deleting collections
in progress. support termlist dump with terms
containing colons.
Merge branch 'diffbot' of github.com:gigablast/open-source-search-engine into diffbot
change deduping logic to be first come first
server, but site rank trumps. fixed bug from
fix before.
added ability to treat <link xyz.com rel=canoical> as meta redirects.
should help us dedup.
added a function to do looser deduping of spider pages although current
not enabled, we are still using the more strict one.
added documentation on how we dedup to developer.html for jon to
take a look at.
added and fixed support for <link ahref=xxx rel=canonical>.
treat those as simplified meta redirects.
updated spider dedup documentation in developer.html file.
doc updates per john on how we dedup
image updates
Merge branch 'diffbot' of github.com:gigablast/open-source-search-engine into diffbot
do not enter into an inifinite keep alive restart loop.
fix infinite keep alive restart bug some more