opened
kevinjpetersen
username_0
Is there any way to improve the memory usage when you need all the bytes into memory? I loaded all the bytes into memory using the "CreateMemoryStreamAsync" and then doing a .ToArray() to convert the bytes into a Base64, so that I can upload this string up to my S3 Bucket.

However, even tho it does work, these are the horrific stats:

An image with the size of **5 MB**, increased the memory usage of the application by **50 MB**.

This happens everytime you select an image, it keeps increasing it. I even tried setting the string to the "SecureString" so it could be disposable, but even that didn't clear the memory afterwards.

So simply, any way to improve the memory usage, or clear the memory usage after not needing it anymore?
Improvement of Memory Usage
issue
created
Tewr
username_1
I'd say no.

The CreateMemoryStreamAsync method should only be used with very small files, for the very reason you are stating.

However, S3 has a multipart upload feature you should use instead. This should keep memory low. This means that you read the file one chunk at a time, like I do in the demo projects, and send each chunk to S3 before you send the next. [this article](https://aws.amazon.com/blogs/aws/amazon-s3-multipart-upload/) lays out the basic principles.
comment
created
Tewr
username_1
Some additional thoughts (still assuming server-side) : I would expect the allocations to be n+(n*1.35) = 2.35n something, for any use case: the byte array in base64 encoding when arriving in dot net land and then the byte array itself. 

In your use-case the byte array will need to be re-encoded into base64 one more time before sending it to S3, so in total n + (2 * 1.35 * n)=3.7n. In your case this should thus theoretically be around 18mb. Sometimes the framework will allocate some more memory in advance, so 50mb is not unimaginable. If you didn't already, try using a memory profiler, this should tell you exactly where the big allocations are.

It may be interesting to expose a pseudo-stream for your use case, avoiding the redundant base64>byte[]>base64 round-trip. Thus the memory footprint should stay at n*1.35. Or perhaps just a more low-level api other than stream directly on top of filereader. I'd like to have some more info on your use case first though (like a profiling session).
comment
closed
Tewr
username_1
issue
created
Tewr
username_1
Closing this as there's no feedback on the issue.
comment