iParadigms/walker — pull request #88 by pchristopher1275 (2014-11-17 22:33:13+00:00)
Title: Content size

This patch addresses the following bullet

* http/ftp/etc. content size limit
In the fetchers we currently do a ReadAll to the response body, but for large responses this is inefficient. We should preallocate a buffer big enough for the claimed response size. We also would like a config parameter to limit the response sizes we'll accept; any response over the max size should act like a failed fetch and indicate in the fetch error that the response was too large for us. One tricky thing is that if the response does not specify a size (content-length), we won't know until we've started reading that the response is too long. This is especially a problem if we try to pass the response to HandleResponse without pre-reading the data; to handle that we would probably have to wrap the reader and make it cut short if the response is too long. For the time being it should be acceptable simply to read the response in every time, like we currently do for HTML pages.