ANUSF/ADA-CMS — Issue #3: ADAPT: Handling of large files
Author: odf
Created: 2011-11-30 23:25:06+00:00
State: open

Adapt is currently not optimised for handling files larger than ca. 50-100 megabytes efficiently. Apart from known issues when uploading via the browser, particularly via low-bandwidth connections, there are limitations within the application code itself. The latter would also affect files that were uploaded to the file share via sftp and then imported into Adapt.

Most importantly, files are currently copied by reading the whole file into memory and writing the contents out to a new file. This obviously limits the size of files that can be handled. At the very least, files should instead be copied in reasonably-sized chunks - say 1MB at a time - in order to reduce memory usage while keeping the number of time-consuming system calls for disk reads low.

An even more efficient solution might be to use the operating systems copying or even renaming functionality wherever possible.

A minor related issue is making sure the local file system is sufficiently dimensioned to hold any temporary files that might be produced during HTTP uploads.