kzk/webhdfs — pull request #4 by jay-altiscale (2013-10-01 00:47:49+00:00)
Title: Accept data stream

current webhdfs gem does not take IO as payload type for PUT and POST request which puts limits on size of data that can be sent over to be less than available memory. This does not work well for big files that needs to be transferred to hdfs.
This patch update the current method in minimal way that takes ruby IO as payload type, that reads the IO in predefined chunk size ( 1024 as hard coded in ruby net http module ), removing any memory limitation.