Ported Fault Tolerant AbstractHDFSExactlyOnceWriter operator from guava poc

     * Added counters to track total time and bytes written.
     * refactored HdfsExactlyOnceOutputOperator to use AbstractHDFSExactlyOnceWriter
     * deprecated AbstractHdfsFileOutputOperator
     * removed obsolete AbstractHdfsOutputOperator
     * removed obsolete AbstractHdfsRollingFileOutputOperator
     * removed obsolete HdfsOutputTupleInterface
     * removed unused benchmark HDFSBothInputOutputOperatorsBenchmark
     * modified HDFSOutputOperatorBenchmark to use fault tolerant writer
     * Updated Fraud detection demo to use fault tolerant writer
     * Updated map reduce demo to use fault tolerant writer
Teardown is not an exception to handling exceptions. None of the code is in fact. Exceptions should never be swallowed.

The code does not assume that any of the operations are unimportant. If any operation fails, flag that there is a failure.

If you want to close as many streams as possible, then you would want to remember the ones which failed, and throw exception after all streams are processed. But then again all those operations need to be done again anyways in most cases due to redeployment. So you probably are better off ditching the container on the first failure by rethrowing the exception.

Another way to think is if its okay if the close fails, then why bother closing it.

Yet another way to think is that app data may now be inconsistent state cause one of the streams could not be closed and the operator swallowed exception.

Does it make sense? Or you want an in person session on Tuesday?

On Oct 5, 2014 7:55 PM, tweise <notifications@github.com> wrote:

In library/src/main/java/com/datatorrent/lib/io/fs/AbstractHDFSExactlyOnceWriter.java:

> +
> +    this.context = context;
> +    lastTimeStamp = System.currentTimeMillis();
> +
> +    fileCounters.setCounter(Counters.TOTAL_BYTES_WRITTEN,
> +                            new MutableLong());
> +    fileCounters.setCounter(Counters.TOTAL_TIME_ELAPSED,
> +                            new MutableLong());
> +  }
> +
> +  @Override
> +  public void teardown()
> +  {
> +    try {
> +      for(FSDataOutputStream outputStream: streamsCache.asMap().values()) {
> +        outputStream.close();


No. In teardown we want to close as many resources as possible. It does not matter if there is a problem with one of the streams or not. What you are referring to would be true for endWindow.

—
Reply to this email directly or view it on GitHub<https://github.com/DataTorrent/Malhar/pull/1110/files#r18440324>.