Would copying them to a different hdfs directory be reasonable (instead of move)? That way, we don't run the risk of losing the data and we also don't affect customer view of things. The downside is we keep two copies of such huge files for the duration of the retention period.

Right now, loading huge history files fails and at the minimum, we need to not have it fail. So given that this is a "temporary" fix, till the real fix being actually processing such files is put in, is that an acceptable solution to mitigate the issue at hand?