Fix some logic in get_caching_iter

Mismatched indentation meant that more chunks than
expected could be yielded.

Make both filesystem ChunkedFile and S3 ChunkedFile
handle the case where the filepointer no longer exists.

Added unit tests which reproduced the problem
using both a filesystem and S3 ChunkedFile.

Fixes bug 1194929.

Change-Id: I0f70155e59f6bc754c88b5b9e96645459bfe770a