(Updating as per my chat with Joep). I thought about skipping files. The rationale about moving them to a different location on hdfs was, after the retention period is over, we would no longer have the file on hdfs if we were to skip them and leave them as is. So we lose the data forever since we couldn't load it in hbase either. But if we move them to a different hdfs location, we at least have the files to process later.

With regular sized files, even if they get deleted after retention period, we know we have a copy of those in job history raw table, so we can reclaim the original file from the hbase table if need be.