cloudsidekick/tclcloud — Issue #2: Patch to reduce memory use when handling large S3 data
Author: samoc
Created: 2012-03-05 12:19:49+00:00
State: open

The patch below removes comparison of $data to the empty string "".
See http://wiki.tcl.tk/Shimmering

$data could contain a large file from S3 (e.g. 300MB + in my application).
A Tcl variable that contains 300MB of binary data consumes 300MB of RAM...
unless you ask Tcl to treat it as a string, in which case a duplicate of the data is created for the internal string representation... and now it takes 600MB.
The comparison of $data to a string "" doubles the memory requirement for $data.

In the code below, there is a happy coincidence that: ($method == "PUT") === ($data != "").
In general, to test for emptiness of a large binary variable something like this can be used:

proc is_empty {data} {
# Is data empty?

set c {}
binary scan $data c c
if {$c == {}} {
return 1
} else {
return 0
}
}

Cheers,

Sam

diff --git a/tclcloud.tcl b/tclcloud.tcl
index 3df7e24..25a4659 100644
--- a/tclcloud.tcl
+++ b/tclcloud.tcl
@@ -315,7 +315,7 @@ proc tclcloud::Perform_query {url header {method GET} {data
if {$timeout != 0} {
lappend cmd -timeout [expr {$timeout * 1000}]
}
- if {$data != ""} {
+ if {$method == "PUT"} {
lappend cmd -type $contenttype -query $data
}
catch {set token [eval $cmd]} error_code