- Fix from Doğacan Güney to get http.useHttp11 property from config

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@441379 13f79535-47bb-0310-9956-ffa450edef68
- NUTCH-354, Stefan's unit test - it passes

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@441382 13f79535-47bb-0310-9956-ffa450edef68
Fix an NPE when using searcher.max.hits, but NOT using time limit.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@447363 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-105 - Network error during robots.txt fetch causes file to beignored, contributed by Greg Kim

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@447893 13f79535-47bb-0310-9956-ffa450edef68
fix for fetcher testcase

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@447931 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-367 - DistributedSearch thown ClassCastException

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@447940 13f79535-47bb-0310-9956-ffa450edef68
Refactor URLNormalizers (NUTCH-365). Iterative normalization has been
implemented, but is not used by default.

Development of this functionality was supported by SiteSell Inc.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449088 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-332: fix the problem of doubling scores caused by links pointing
to the current page (e.g. anchors).


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449102 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-336: differentiate between newly discovered pages (known value through
inlink contributions) and newly injected pages (aribtrarily defined initial
value).


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449274 13f79535-47bb-0310-9956-ffa450edef68
In case we fail to filter the score, bring it to 0.0f (and remove unused
variable).


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449278 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-337: obey "fetcher.parse" property if -noParsing is not specified.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449287 13f79535-47bb-0310-9956-ffa450edef68
Fix JUnit test after internal API change.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449288 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-350: urls incorrectly marked as STATUS_FETCH_GONE when blocked by
http.max.delays. Instead the status is set to STATUS_FETCH_RETRY. Since this
is an intermittent problem related to the Fetcher implementation, we don't
increase the retry counter.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449293 13f79535-47bb-0310-9956-ffa450edef68
updated tutorial and news, republished whole site because of changes in navigation

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449376 13f79535-47bb-0310-9956-ffa450edef68
Don't create dummy Content (throws NPE), just pass null. Reported by
Richard Braman.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449738 13f79535-47bb-0310-9956-ffa450edef68
Catch exception on invalid urls, and continue collecting valid ones.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@449752 13f79535-47bb-0310-9956-ffa450edef68
Bring back the '-noAdditions' option. This is useful for running
constrained crawls, where the complete list of URLs is known in
advance.


git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@450799 13f79535-47bb-0310-9956-ffa450edef68
cleaned up package to use generics and java5 loops, removed unnecessary log guards, fixed indentation

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@451640 13f79535-47bb-0310-9956-ffa450edef68
cleaned up package to use generics and java5 loops, removed unnecessary log guards, fixed indentation

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@451641 13f79535-47bb-0310-9956-ffa450edef68
NUTCH-374: when http.content.limit be set to -1 and Response.CONTENT_ENCODING is gzip or x-gzip , it can not fetch any thing.(King Kong)

git-svn-id: https://svn.apache.org/repos/asf/lucene/nutch/trunk@451648 13f79535-47bb-0310-9956-ffa450edef68