Unidata/IDV — pull request #79 by tommyjasmin (2014-06-27 16:47:46+00:00)
Title: Fix buffer bug, reads greater than 2 GB were failing [1483]

Hi guys - very important this one gets in. Definite buffer overflow issue in IOUtil.writeTo()
The counter was an int and of course that means our read/write limit is 2 GB. I had to track
it down because MUG reported Suomi M-BAND bundles worked fine, but I-BAND bundles
would fail to save and reload. The only obvious difference to me was size (I-BAND granules
are MUCH bigger).

Watch the writeTo logging I added overflow Java signed int limits:

11:56:39.193 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: Transferred 2147300 Kbytes
11:56:39.194 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: num read: 100000
11:56:39.194 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: Transferred 2147400 Kbytes
11:56:39.195 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: num read: 100000
11:56:39.195 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: Transferred -2147467 Kbytes
11:56:39.196 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: num read: 100000
11:56:39.196 [AWT-EventQueue-0] TRACE e.w.ssec.mcidasv.PersistenceManager - writeTo: Transferred -2147367 Kbytes

The negative total returned by the method is considered a failure by callers.
Long story short (twisted pun, sorry), cumulative count needed to be a long.
Luckily the ripple to other code was not bad.

public static long writeTo(InputStream from, OutputStream to,
Object loadId, long length)
throws IOException {
from = new BufferedInputStream(from, 100000);
byte[] content = new byte[100000];
long total = 0;
while (true) {
int howMany = from.read(content);
if (howMany <= 0) {
break;
}
total += howMany;
if ( !JobManager.getManager().canContinue(loadId)) {
return -1;
}
String msg;
if (length > 0) {
msg = "Transferred " + (total / 1000) + "/" + (length / 1000)
+ " Kbytes ";
} else {
msg = "Transferred " + ((total) / 1000) + " Kbytes";
}

JobManager.getManager().setDialogLabel2(loadId, msg);

to.write(content, 0, howMany);
}
return total;
}