chimera: change FileSystemProvider#move() return value meaning

FileSystemProvider#move() always returns true. Instead of changing
the interface let update the meaning and return true if underlying filesystem
was change and false if not. This is help to control attribute caching.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
src: split nfs code info a dedicated module

we quite often need to update NFS code and it's much
simple if a new jar file contains only nfs changes.

There is a change, in pinning via nfs: it will
use DcapProtocolInfo to avoid dependency loop.
Matches current SRM behaviour.

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
spacemanager: Set file to TRANSFERRING on accept file request message

By changing the file entry to TRANSFERRING on the accept
file request rather than response we save:

- A database lookup
- A database update
- A message queueing phase

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6547/
spacemanager: Replace freespaceinbytes with availablespaceinbytes in link group table

Queries and triggers become simpler when we track available space rather
than free space.

On upgrade, the link group table will be updated. The table is small and the
schema change will be almost instantaneous. Note however that should dCache
be downgraded, the schema changes have to rolled back using one of the 'dcache
dabase rollback' commands.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6532/
pool: Bind FTP data channel to single interface

The FTP protocol only allows a sinle IP address to be returned to the client,
yet the mover listened on the wildcard address.  This has lead to surprising
results, as an application can by a socket to the wildcard address even when
another socket is already bound to the same port on a specific interface. This
could lead to situations in which the client would connect to the other socket
and not the socket created by the mover (e.g. the admin door that listens in
the same port range).

The patch addresses the problem by only binding the server socket to the
address we would return to the client. The patch changes the behaviour by
removing one DNS reverse lookup and DNS forward lookup: Previously the pool
would return the canonical name of the interface and the door would resolve
that back to an IP address before returning the address to the client. Now the
pool send an address to the door and the door just passes on that address. In
theory this could lead to changes in which pool interface is reported. For this
reason I do not request this patch to be merged to stable branches.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6593/
system-test: Fix wrong path in hostCredentials script

Fixes a recently introduced regression.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6604/
chimera: Optimize read of file and directory attributes

Chimera's inode class provides convenient access to the attributes stored in
the inode table. It also supports caching that information, which allows it to
be used in modular code without repeated loading of the same database row.

Chimera does not provide such support for attributes not stored in the inode
table, eg locations, access latency, retention policy and tags. Those have to
be queried through the FileSystemProvider interface. Besides not being
particularly OO like, this approach lacks the convenient caching feature of the
inode.

This patch introduces the dCache specific ExtendedInode subclass of FsInode.
It provides access to those attributes that are not stored directly in the
inode table. Those attributes are cached, which means the ExtendedInode class
is primarily intended for reading an inode, not for updating it.

In particular access to locations and tags has been optimized. Previously
locations could only be read for a particular location type, requiring two
queries to the database. This cost of reading all locations at once is likely
to be very close to that of just reading locations of one type, so this patch
adds means through which to read all locations in a single query. Likewise for
tags: Existing code would always first check for the existence of a tag by
loading the entire list of tag names for a directory. It would then for each
tag do another lookup in the same table to read the tag id, followed by a read
in another table to load the data. This has now all been collapsed to a single
method capable of reading all tag data in one operation by joining the two
tables.  The data is cached in the ExtendedInode, which allows those tags to be
access by the storage info extractor without additional queries.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6611/
srmclient: fix support for URLs with IPv6 address

a bit of DRY, but have to be solved more globally.

Ticket: #8248
Acked-by: Dmitry Litvintsev
Target: master
Require-book: no
Require-notes: no
pool: fix lost space messages (and cause)

Since version 5.0.84 Berkeley DB generates statistics files like je.stat.csv,
which make pool space calculation unhappy. As we don't use this
information, disable to avoid annoying messages about lost space
(even in case of read-only activities):

---
26 Feb 2014 16:36:55 (dcache-lab002-A) [] The file system containing the data files appears to
have less free space (6,094,454,210,560 bytes) than expected (6,094,454,214,656 bytes);
reducing the pool size to 16,435,147,381,007 bytes to compensate. Notice that this does not
leave any space for the meta data. If such data is stored on the same file system, then it is paramount
that the pool size is reconfigured to leave enough space for the meta data.
---

Acked-by: Gerd Behrmann
Target: master
Require-book: no
Require-notes: no
srm-client: fix adler32

The adler32 script invokes the Adler32 class to calculate the ADLER32 digest of a file.

Subsequent to the 2.6.0 release, this class was mistakenly moved to srm-server, so isn't
part of the srm-client packages, which means that this command fails if srm-client is
built from branches newer than 2.6.

This patch moves the Adler32 class to the srm-client package.  It also moves the GridftpList
class that is also used by a script (which we currently don't distribute).  The two scripts
command-line-clietn and srm_v1 (both of which we don't distribute) are deleted as they
refer to classes that do not exist.  The unused Makefile is also deleted.

Target: master
Request: 2.8
Request: 2.7
Acked-by: Gerd Behrmann
Patch: http://rb.dcache.org/r/6643/
use ISO 8601 time format in sweeper ls -l output

"sweeper ls -l" output did not contain year in the format
of creation and last access timestamps. This patchs modifies
ouptut to show time in ISO 8601 time format :

admin > sweeper ls -l
  0 000029A61FE585D94C74955AE61BF3AB396F  CACHED          993 2014-02-21T17:47:37 2014-02-21T17:47:37
  1 00009C6579BE112B40848D2F1221CB5D03FC  CACHED          993 2014-02-21T21:36:57 2014-02-21T21:36:57

RELEASE NOTES:

"sweeper ls -l" output did not contain year in the format
of creation and last access timestamps. This patchs modifies
ouptut to show time in ISO 8601 time format :

admin > sweeper ls -l
  0 000029A61FE585D94C74955AE61BF3AB396F  CACHED          993 2014-02-21T17:47:37 2014-02-21T17:47:37
  1 00009C6579BE112B40848D2F1221CB5D03FC  CACHED          993 2014-02-21T21:36:57 2014-02-21T21:36:57

Patch: http://rb.dcache.org/r/6634/
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan
Target: trunk
poolmanager: Add partition for transfer buffers

Adds the 'buffer' partition type. This partition type is for pools that are
used as temporary transfer buffers rather than long term storage.

Pool selection does not take free space into account (with the exception of not
selecting full pools for writes). Pools are selected by load only, using a
randomized weighted selection with 1/2^(cc * mc * n) as the weigth, where n is
the number of movers on the pool, mc is the mover cost factor of the pool and
cc is the cpucostfactor as configured in the partition. Mover queue limits do
not influence pool selection.

Buffer pools do not support hot spot replication, cost based link fallbacks,
replication limits, same host replication protection, nor idle cost cuts.
Buffer pools do support pool to pool transfers as well as stage from tape,
although the current implementation may not be well suited for tape pools.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
srm,srm-client: Report failure for failed SRM v1 transfers

Fixes the following bugs:

- The srm client ignored failures to set the file status to 'Done'.

- The srm client changed the file status to 'Done' even when the
  transfer failed.

- The srm server did not accept 'Failed' as a file status even
  though the spec lists it as a valid status for setFileStatus.

I am requesting a merge to stable branches as this bug may report
failed uploads as having been successful.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6629/
chimera: Add mv to chimera shell

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6632/
Update to liquibase 3.1.1

I need the new beforeColumn attribute in subsequent space manager patches.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6635/
loginbroker: Register root path of doors

The patch adds the root path of each door to the information registered in
loginbroker.

The following properties are obsolete:

srm.service.webdav.root
srm.service.xrootd.root

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6641/
chimera: Minor refactoring of tag reading code

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6642/
Convert tabs to space
nfs: remove unused exception declaration

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
nfs: fix NPE on door reboot

handle PoolPassiveIoFileMessage message comming from a pool
even if door did not initialize it. Happend on door restarts
during heavy IO load.

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no