fixed deleted files error

if flag files are deleted during file collection the process would continue
with an invalid list of files and ultimately create an empty archive.
This patch adds an additional check after the file collection that leaves the
script if the list was corrupted by deleted files.
adjust indent in pack-files scripts
notebook cleanup and some small fixes
simplify file collection algorithm

remote verify step as it does not really prevent problems with deleted flags.
Instead, an additional check now checks if the whole collection succeeded and
stops the process if the script was hit by a deleted flag.

The detect deleted method is now included in the collectFiles method. This way
we avoid double requests to chimera.
reduce log output of hsm-internal-* scripts (put only)

the scripts were reporting to much details. This patch removes the status
messages and only leaves the error messages.
Merge remote-tracking branch 'origin/2.6' into 2.6
fix pack-remaining feature

while looking for recently changed files, the script would always
find its own lock file and thus not create a archive. This is
fixed by this patch.
use find name filter rather than regex for recent file
use zip rather than tar

tar reads the file up to the position the file to extract is located.
zip on the other hand has a header that allows direct access.
This patch changes the archiving method to zip.
traverse directory recursively when zipping files

otherwise the metadata file would be ommitted
consolidate packing scripts

the two packing scripts share most logic. The only difference is
in obtaining the group directories and how to extract the tags from
this directory.

This patch merges the two scripts and has the differences extracted
in two scripts containing the mode specific functions. The mode can
be chosen by passing by-group.sh or by-dir.sh as first parameter to
the master script pack-files.sh.
fix missing quote
fix getGroupSubDir

getGroupSubDir did not return anything. This patch adds the missing echo
statement.
consolidate hsm-internal scripts

with this patch there is only one script for all archiving modes:
hsm-internal.sh
The script gets a new parameter packMode that has to set to point to the
corresponding hsm-by-dir or hsm-by-group script that will be sourced by
the hsm-internal script and be used for performing the mode specific
tasks.
use mongoDb instead of chimera-cli in hsm-script
add missing datasetPut.js mongodb script file
fix mongo db parameter passing
remove superfluous archiveUrl.

It is sufficient if this is added by the packing node as soon as needed.
add container.conf

The container.conf file is the global configuration containing
general options like the hsm subdirectory, database url, etc
and sections for directories that contain small files together
with specific options.
update dcache