[L38-4] NNStorage abstraction and storage inspector

Summary:
This patch introduces changes to FSImage and separates the
abstraction
of NNStorage, and FSImagePreTransactionalStorageInspector. Once we
have
L38, there will be another inspector implementation for transaction
based edit logs.
However, the pre-transactional one will stay for upgrade purposes.

Also NNStorage contains some hooks for the new layout, not used
currently, e.g., get edit file name based on txids.

Test Plan:
ant test, all test cases for upgrade/rollback/loading
namespace/corrupted storage dirs/save namespace...

Reviewers: hkuang, pritam, avf

Reviewed By: hkuang

Differential Revision:

Task ID: 1009397

Blame Rev:
[L38-5] Introduce MD5 utilities

Summary:
With the L38 layout, the md5 of each image is stored in a separate
file.
We can no longer store md5 in version file, since we can have
multiple
images in a storage directory.

This patch introduces utilities for writing a computed md5 into
a file.

AtomicFileInputStream is a class to guarantee that the hash
is written
atomically (using temporary file, and rename).

MD5FileUtils is a class for writing, reading, and
validating md5
digests.

Test Plan: added two test suites

Reviewers: hkuang, avf

Reviewed By: avf

Differential Revision:

Task ID: 1009397
[L38-6] Introduce new layout storage inspector

Summary:
This patch introduces the inpector for L38 storage directories, it's
a
counterpart for FSImagePretransactionalStorageInspector. This
inspector
will be used once we bump up the layout number.

Test Plan: added test case for the inspector

Reviewers: hkuang, pritam, avf

Reviewed By: avf

Differential Revision:
Addressing TestFileCreation j-unit case failure from D563060

Summary: Addresses TestFileCreation j-unit failure from previous commit by moving the lock back to the top. Shouldn't have to worry about the read in the lock since it only happens during the initialization of the datanode.

Test Plan:
Ran J Unit Test TestFileCreation

Some of the Junit results that were failing

Reviewers: weiyan

Reviewed By: weiyan

Blame Revision: D563060
Add guava dependencies for contrib/HA

Test Plan: ant clean compile

Reviewers: avf, hkuang

Reviewed By: avf

Differential Revision:
s patch does not touch the avatar side. I did not run all unit test, so it's a working copy.

Test Plan: ported a whole bunch of unit tests

Reviewers: hkuang, avf

Reviewed By: avf

Differential Revision:
Summary:
This patch introduces the abstraction of JournalManager, JournalSer,
RemoteEditLog, RemoteEditLogManifest.

It also provides the initial implementaion of FileJournalManager.
The implementations are not connected to the current code, I am just
putting them in place.

Test Plan:
added test case for retention manager, other test cases will be
added
with the layout bump

Reviewers: hkuang, avf

Reviewed By: avf

Differential Revision:

Task ID: 883594

Blame Rev:
[L38-8] Configuration changes

Summary:
This patch enables to configure edits/image directories by
specifying
uri to the storage. file:/ prefix is to be used to specify
FileJournalManager. Bookkeper can be specified by bookkeeper:/...

By default, uris without specified schema are treated as file
journals/image directories.

This patch temporarily changes Avatar initialize function(),
once we
introduce new layout, this will change. For now, if the
configuration
specifies something different than files, the avatar
initialize will
crash.

-removed some code from Avatar identical to NameNode.

Test Plan: ant test

Reviewers: hkuang, avf, pritam

Reviewed By: avf

Task ID: #

Blame Rev:
[L38-9] Change http interfaces to handle new layout

Summary:
This patch restructures image/edits uploading/downloading, and
prepares
those classes for transactional layout.

It seems that the secondary NN was not validaing MD5 hash by
now. Fixed
this.

Test Plan:
ant test, manual

added new test cases

Reviewers: hkuang, avf

Reviewed By: avf

Differential Revision:

Task ID: 1009397
Fix build SnapshotNode imports

Summary:

merging new layout fix

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
HDFS: fix test bug in TestTransferBlock

Summary:
TestTransferBlock doesn't fail the test when needed. Fix it. Test still passes after the fix.
Also, use the more commonly used DFSTestUtil.createFile() to create test files.

Test Plan: run the test
FsShellService: Fix bugs

Summary:
1. listStatus: handle non-existing directory correctly.
2. fix start-fsshellservice.sh and stop-fsshellservice.sh
3. Change all Fsshellxxx names to FsShellxxx.

Test Plan: manual tests

Reviewers: pkhemani, weiyan, hkuang

Reviewed By: weiyan
Rethrow the exception

Summary:
We need to rethrow the exception to preserve the old behaviour.
Right now when the error happens we do not actually handle it sometimes.

Test Plan: unittests and the test cluster

Reviewers: aching

Reviewed By: aching

CC: mattwkelly
Fix re-replication in BlockPlacementPolicyConfigurable.

Summary:
We want to fallback to the default block placement policy for
re-replication since BlockPlacementPolicyConfigurable causes too much
traffic to a small set of nodes during re-replication.

Test Plan:
1) All unit tests.
2) Enhanced TestReplication

Revert Plan:

Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: kannan, hkuang, sdong, weiyan, tomasz

Reviewed By: hkuang

Task ID: 1374530
[HDFS Build] fix build_all failure by excluding namespace-notifier build.xml

Summary:
Since Tom checked in the new layout changes, namespace-notifier codes are not compatible.
And it seems there is no easy fix for that. Temporarily exclude it to unblock build_all.sh

Test Plan: run build_all

Reviewers: sdong

Reviewed By: sdong
Fixing probably-missed sync in VolumeMap.update

Summary:
It looks like the update method of VolumeMap
has to be synchronized to be consistent.
Is this the case?

Test Plan: sanity-check

Reviewers: tomasz, hkuang, sdong

Reviewed By: sdong
[L38-10-bump] Plugin JournalManager to hdfs-core

Summary:
This patch plugs in the JournalManager interface into
FSEditLog/FSImage.
It only changes the core hdfs classes.

Compile:

ant compile-core
ant compile-core-test

This patch does not touch the avatar side.
I did not run all unit test, so it's a working copy.

Test Plan: ported a whole bunch of unit tests

Reviewers: hkuang, avf

Reviewed By: avf

Differential Revision:
[L38-11-elopt] Port edit log optimizations.

Summary:
With this patch, we are going to have substantial savings on writing
to
edit log, as all longs, shorts, will be written in binary form
without
expensive transformations into utf8 strings (coding,decoding). Also
for
each operations, we will not be writing the length of writables.
This
change is compatible with Apache image layout -39 -> in our case it
will
be -40, since we have a shift for hardlinks.

Test Plan: will run ant-test-core

Reviewers: hkuang, avf

Reviewed By: avf

Differential Revision:

Task ID: 883594

Summary:

Test Plan:

Reviewers:

CC:

Task ID: #

Blame Rev:
[L38-12-bumpavatar] Plugin journal manager into avatarnodes

Summary: This patch changes the standby mechanism of ingesting edits,
and plugs in the journal manager interface.With this patch we have a
full flavour implementation of journal manager, the standby is doing
checkpoints, and also is able to failover.

****I added mechanism for recovering from ingest failure, after several
failures, the state of stanby is reset, and we attempt to recover from
problems

****With this patch, it is possible to save namespace when standby is
up, and standby should proceed in this situation, so there is no need
for handling stale checkpoints etc.

----------------------------------------------

Problems:

1) What should happen when we are quiescing standby to -1 transaction
id. Since I try to obtain the streams gradually with retries, the last
stream that does not exist will cause 30 retries before the standby
quiesces

2) The biggest problem that I see now is with the catchUp concept. For
now it's implemented in a way where ingest looks if it's consuming the
inprogress segement. If so, it checks the size of the stream and the
current position and compares them. If it's consuming a finalized
segment, we assume that it's caching up. This means that BK
EditLogInputStream must implement lenght() method for in progress
segments at least, which would give byte size of the stream.

----------------------

All tests, including new ones work, but I'm still working on this patch.
Added tests for
-ingest recover
-reprocessing ingest
-retries when obtaining input stream

--------------------

I removed snapshot from compiling, because the new layout is not there
at all!! I also made some changes in OfferService.java and Client.java,
which are in the trunk, which fixed the timing out test cases.

Test Plan: ant test, manually new test cases

Reviewers: avf, hkuang

Reviewed By: avf

    Task ID: 883594
Fix for the SocketTimeoutException

Summary:
My patch yesterday was wrong. We still need to handle it as any
IOException, with the only exception. This fixes it, we are only
changing the cleanup behaviour, not the rest of the error processing.

Test Plan: deploy and run

Reviewers: aching, rvadali

Reviewed By: rvadali