tenderlove/mechanize — Issue #211: Mechanize::Download does not stream direct to file
Author: krisb
Created: 2012-03-02 20:51:52+00:00
State: open

I've tried to use mechanize to download some large files and noticed that the file was not being written to whilst downloading and the process memory was increasing. I've followed the instructions in using Mechanize::Download and I've included a sample script to illustrate the problem. I tried a couple of variations in the calls to Mechanize to try to get this working.

Looking over the code, it looks like the body is being read prematurely but I'm not sure of the best way to skip this happening without affect the other parsers. I'm using ruby 1.9.3-p0 and mechanize 2.3. I tried mechanize 2.2.1 also and this has the same behaviour. max_file_buffer and max_history appear to have no affect

```ruby

#!/usr/bin/env ruby
require 'fileutils'
require 'logger'
require 'rubygems'
require 'mechanize'

samples = ["http://releases.ubuntu.com/oneiric/ubuntu-11.10-desktop-i386.iso"]

class Downloader
def initialize
@log = Logger.new $stderr
# Mechanize.log = @log
@agent = Mechanize.new
@agent.pluggable_parser.xml = Mechanize::Download
@agent.pluggable_parser.default = Mechanize::Download
@agent.max_file_buffer = nil
@agent.max_history = 0
end

def filename_from_uri(uri)
(URI(uri).path.split("/") || []).last || "unknown"
end

def download_file_v1(uri, file = filename_from_uri(uri))
@log << "get #{uri}\n"
resp = @agent.get(uri)
@log << "resp class: #{resp.class}\n"
@log << "save #{uri} -> #{file}\n"
resp.save(file)
@log << "downloaded #{file}\n"
end

def download_file_v2(uri, file = filename_from_uri(uri))
@log << "download #{uri} -> #{file}\n"
resp = @agent.download(uri, file)
end
end

downloader = Downloader.new
samples.each {|url| downloader.download_file_v2 url }
```