Update branch ready for the 2.9 branch
utils: move Args class into commons

An old copy of the Args class exists in srm-common package.  This patch
moves the Args class from cells to dcache-common so it may be used by
SRM and dCache code alike.

The patch removes Serializable from Args.  This is because we currently don't
use Args in Messages and want to discourage any future use.

The patch also updates the import section of the files affected to follow
the agreed ordering.

Target: master
Requres-book: no
Requires-notes: no
Patch: http://rb.dcache.org/r/6506/
Acked-by: Gerd Behrmann
srm-client: switch to common Args

Update srm-client to use the now common Args class

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/6522/
Acked-by: Gerd Behrmann
cells: remove conflict between CellAdapter and CellShell

The 'say' and 'exec context' commands exist both in CellAdapter and
CellShell.  This is normally not a problem unless an instance of
both classes share the same command context, which happens for the System
cell.

This patch removes the conflict by dropping the two commands from
CellAdapter.  The effect of doing this is to remove these two commands
from cells other than System.

Neither command is used by dCache outside of the System cell (or some
other CellShell), so dCache itself will not be affected by these changes.

It's possible, but I think unlikely, that our admins will use either
command.

FOR RELEASE NOTES

The 'say' and 'exec context' commands are no longer available from
cells other than the System cell.

Target: master
Requires-notes: yes
Requires-book: yes
Patch: http://rb.dcache.org/r/6523/
Acked-by: Gerd Behrmann
ftp: Reenable support for RFC 2428

To avoid that IPv6 enabled clients use extended passive and extended port
commands on IPv4 connections, support for these commands is limited to
IPv6 connections. This is necessary as using these commands prevents
ftp data connections from being redirected to pools.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/r/6525/
dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape Transfers page)

The Wicket PropertyModel requires either an object field name or a getter/setter
name which can be used to invoke by reflection in order to store and retrieve the property value.

In the case of a ListView, the PropertyModel value is a list; in the case of the TapeTransfersQueue page,
this list is provided by a service whose DAO abstraction accesses a page cache that is refreshed
periodically by calling "xrc ls" on the PoolManager.  The current implementation uses a getter
which extracts a new copy of the list from cache.

This getter, however, is invoked each time the "populateItem" method is called (that is, for each list item);
aside from being inefficient, this could cause an indexing problem if the size of the underlying
list changes in the middle of the procedure to populate the entire list; specifically, the
reported bug suggests that either the original or the current list size is used to determine
the successive iteration, but on that successive call, the list size has been reduced (by one),
causing an IndexOutOfBounds exception.

The solution here is to associate a stable instance of the list with the page constructor.
There is no problem in terms of updating the page values, since this particular page is only
refreshed manually by reloading. A fresh copy of the list will be obtained from the cache on
each reload, which calls the page constructor.

Testing:

Before this patch, loading the page causes the browser to hang and
the following stack trace is observed in the log:

1.04.37 [qtp1040303644-45] [] Unexpected error occurred
org.apache.wicket.WicketRuntimeException: Error attaching this container for
rendering: [Page class = org.dcache.webadmin.view.pages.tapetransferqueue.TapeTransferQueue, id = 0, render count = 1]
...
Caused by: java.lang.IndexOutOfBoundsException: Index: 9739, Size: 9739
at java.util.ArrayList.rangeCheck(ArrayList.java:604) ~[na:1.7.0_25]
at java.util.ArrayList.get(ArrayList.java:382) ~[na:1.7.0_25]
at org.apache.wicket.markup.html.list.ListItemModel.getObject(ListItemModel.java:58) ~[wicket-core-1.5.10.jar:1.5.10]

After patch application, the error no longer occurs.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6527
Require-book: no
Require-notes: yes
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8225
Acked-by: Tigran

RELEASE NOTES:  Fixes a bug in the TapeTransferQueue page whereby an IndexOutOfBounds error
can occur (and hang the page).
dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape Transfers page)

The Wicket PropertyModel requires either an object field name or a getter/setter
name which can be used to invoke by reflection in order to store and retrieve the property value.

In the case of a ListView, the PropertyModel value is a list; in the case of the TapeTransfersQueue page,
this list is provided by a service whose DAO abstraction accesses a page cache that is refreshed
periodically by calling "xrc ls" on the PoolManager.  The current implementation uses a getter
which extracts a new copy of the list from cache.

This getter, however, is invoked each time the "populateItem" method is called (that is, for each list item);
aside from being inefficient, this could cause an indexing problem if the size of the underlying
list changes in the middle of the procedure to populate the entire list; specifically, the
reported bug suggests that either the original or the current list size is used to determine
the successive iteration, but on that successive call, the list size has been reduced (by one),
causing an IndexOutOfBounds exception.

The solution here is to associate a stable instance of the list with the page constructor.
There is no problem in terms of updating the page values, since this particular page is only
refreshed manually by reloading. A fresh copy of the list will be obtained from the cache on
each reload, which calls the page constructor.

Testing:

Before this patch, loading the page causes the browser to hang and
the following stack trace is observed in the log:

1.04.37 [qtp1040303644-45] [] Unexpected error occurred
org.apache.wicket.WicketRuntimeException: Error attaching this container for
rendering: [Page class = org.dcache.webadmin.view.pages.tapetransferqueue.TapeTransferQueue, id = 0, render count = 1]
...
Caused by: java.lang.IndexOutOfBoundsException: Index: 9739, Size: 9739
at java.util.ArrayList.rangeCheck(ArrayList.java:604) ~[na:1.7.0_25]
at java.util.ArrayList.get(ArrayList.java:382) ~[na:1.7.0_25]
at org.apache.wicket.markup.html.list.ListItemModel.getObject(ListItemModel.java:58) ~[wicket-core-1.5.10.jar:1.5.10]

After patch application, the error no longer occurs.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6527
Require-book: no
Require-notes: yes
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8225
Acked-by: Tigran

RELEASE NOTES:  Fixes a bug in the TapeTransferQueue page whereby an IndexOutOfBounds error
can occur (and hang the page).
(cherry picked from commit ed919cf0eb99da85d594ebbc4d27fba6b1c9d09e)

Signed-off-by: alrossi <arossi@otfrid.fnal.gov>
chimera: Optimized creation time column changeset

After extensive tests, I found a way to significantly improve the time it takes
to add the creation time column.

The key problem is not adding the column but to update every row in the
t_inodes table to populate the new column. Due to the MVCC nature of postgresql
tables, updating a row doesn't replace it - instead a new copy of the row is
added while the existing row is kept until the transaction within which the
update executes is committed. Since every row in the entire table is updated
in a single transaction, the table effectively doubles in size during this
process. In general, when a row is updated, any index pointing to the row needs
to be updated too. This was the source of the high amount of I/O observed on
our postgresql instance.

Postgresql has a trick to reduce the cost of updates: Heap-only tuples (HOT)
allow the updated row to be stored within the same logical block as the
existing row. In such cases the index doesn't need to be updated, as the index
really points to blocks, not individual rows. HOT updates however require that
there is enough free space in blocks to accomodate additional copies of rows.
During regular operation, deleted rows leave such free space. Also updated rows
once the transaction is committed leave the old copy free to be reclaimed.

In this particular update we however update every record in the entire table
within a single transaction. This means that to avoid expensive index updates,
we need enough free space in blocks to duplicate every row. This is exactly
what this patch achieves: It changes the fillfactor (percentage of a block
filled with regular inserts, leaving the rest available for HOT updates) to
45%, leaving plenty of room for adding an additional copy of every row. After
the update the fillfactor is raised to 75% to avoid wasting too much disk space
in the future.  The default fillfactor in postgresql is 100%, which is fine for
mostly static data, but not fine if rows are updated frequently (we update
inodes frequently due to atime updates and due to link count changes for
directories). The addition of the new column is altered too: We now add the
non-null constraint with a computed default value right away, forcing a table
rewrite (which will use the new fill factor). Once the transaction is
committed, old copies of rows can be reallocated, so there will again be 55%
free space in the table.  Note that after raising the fillfactor to 75%, we do
*not* rewite the table to reclaim the space. Postgresql will use the space when
creating new inodes, but an admin can manually reclaim the space by running
vacuum full on t_inodes (only recommended with Postgresql 9.0 or higher - use
cluster on older versions). Unless tight on disk space, this should not be
necessary.

The difference of this change on our test system is tremendous (similar hardware
to our production system, except significantly less RAM). The old changeset took
approximately 41 hours to complete (for around 48 million inodes - the same
changeset took approx 5 hours on our production hardware). The updated changesets
complete in approx. 40 minutes (half of that is for adding the column; the other
half for update the creation time).

The changeset has been written such that it will not run if the column was already
added to t_inodes (as is the case at NDGF).

Target: trunk
Request: 2.8
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6530/
chimera: Optimized creation time column changeset

After extensive tests, I found a way to significantly improve the time it takes
to add the creation time column.

The key problem is not adding the column but to update every row in the
t_inodes table to populate the new column. Due to the MVCC nature of postgresql
tables, updating a row doesn't replace it - instead a new copy of the row is
added while the existing row is kept until the transaction within which the
update executes is committed. Since every row in the entire table is updated
in a single transaction, the table effectively doubles in size during this
process. In general, when a row is updated, any index pointing to the row needs
to be updated too. This was the source of the high amount of I/O observed on
our postgresql instance.

Postgresql has a trick to reduce the cost of updates: Heap-only tuples (HOT)
allow the updated row to be stored within the same logical block as the
existing row. In such cases the index doesn't need to be updated, as the index
really points to blocks, not individual rows. HOT updates however require that
there is enough free space in blocks to accomodate additional copies of rows.
During regular operation, deleted rows leave such free space. Also updated rows
once the transaction is committed leave the old copy free to be reclaimed.

In this particular update we however update every record in the entire table
within a single transaction. This means that to avoid expensive index updates,
we need enough free space in blocks to duplicate every row. This is exactly
what this patch achieves: It changes the fillfactor (percentage of a block
filled with regular inserts, leaving the rest available for HOT updates) to
45%, leaving plenty of room for adding an additional copy of every row. After
the update the fillfactor is raised to 75% to avoid wasting too much disk space
in the future.  The default fillfactor in postgresql is 100%, which is fine for
mostly static data, but not fine if rows are updated frequently (we update
inodes frequently due to atime updates and due to link count changes for
directories). The addition of the new column is altered too: We now add the
non-null constraint with a computed default value right away, forcing a table
rewrite (which will use the new fill factor). Once the transaction is
committed, old copies of rows can be reallocated, so there will again be 55%
free space in the table.  Note that after raising the fillfactor to 75%, we do
*not* rewite the table to reclaim the space. Postgresql will use the space when
creating new inodes, but an admin can manually reclaim the space by running
vacuum full on t_inodes (only recommended with Postgresql 9.0 or higher - use
cluster on older versions). Unless tight on disk space, this should not be
necessary.

The difference of this change on our test system is tremendous (similar hardware
to our production system, except significantly less RAM). The old changeset took
approximately 41 hours to complete (for around 48 million inodes - the same
changeset took approx 5 hours on our production hardware). The updated changesets
complete in approx. 40 minutes (half of that is for adding the column; the other
half for update the creation time).

The changeset has been written such that it will not run if the column was already
added to t_inodes (as is the case at NDGF).

Target: trunk
Request: 2.8
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6530/
(cherry picked from commit b25f8dfbc214d09ae62be3da0a8f16f1c283c612)
Merge pull request #395 from gbehrmann/fix/2.8/rb6530

chimera: Optimized creation time column changeset
ftp: Reenable support for RFC 2428

To avoid that IPv6 enabled clients use extended passive and extended port
commands on IPv4 connections, support for these commands is limited to
IPv6 connections. This is necessary as using these commands prevents
ftp data connections from being redirected to pools.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/r/6525/
Merge pull request #396 from paulmillar/fix/ipv6-ftp

ftp: Reenable support for RFC 2428
Merge pull request #392 from alrossi/fix/2.8/tape-transfer-queue

dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape T...
Include arguments and default values in admin command help output

Also renames a couple of annotation fields to better reflect their
meaning.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6524/
Use annotated command syntax for most shared commands

The patch updates most admin shell commands shared by most services
to the annotated command syntax. This provides nicer help output and
avoid that services like pin manager and space manager show a mixture
of old and new help styles.

The command scanner had to be extended to supported inherited commands.

The help command has been turned into a real command, which means
it will appear in the help output and has a help page too. To avoid
that it gets included multiple times due to being inherited multiple
times, I made the help command optional in the CommandInterpreter.
CellAdapter and UserAmdinShell enable it.

The save and reload commands are now only loaded in services that
actually support these commands.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6531/
system-test: Add utility to start HSQLDB Swing Console

Simplifies manual inspection of the database.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6533/
chimera-nfs: add bringonline command to ".(fset)"

This patch depends on 6409, but also on current trunk/master being fixed (see testing below).  Currently, these changes have not been tested because of the bug blocking their operation.

Adds an operation type to the ".(fset)" dot command to be able to bring a file on line from an archived state.  As with the ".(get)()(locality)" operation, this requires a file system interface in Chimera which is also dCache aware, so that calls can be made to the PoolManager and PinManager.

The following changes have been made:

1.  FileSystemProvider API extended with two methods, pin() and unpin().
2.  PSET inode slightly refactored, with additional call to the filesystem to set pinned state.
3.  PinManager cell endpoint added to DCacheAware filesystem, in order to implement the two added methods.
4.  Relevant timeout properties for the PinManager instance added.

TESTING:

on testbed, using 4.1.  Pinning and unpinning, along with lifetime, works as it should.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6410
Require-notes: yes
Require-book: yes
Depends-on: http://rb.dcache.org/r/6409

BOOK:
RELEASE NOTES:

A new command has been added to the NFS dot-command set to allow users to pin and unpin files.  Files which are not on disk will be brought on line.

The command takes the following form:

            touch ".(fset)(filename)(pin|stage|bringonline)(duration)([SECONDS|MINUTES|HOURS|DAYS])"

The variants of the third argument are equivalent in effect.

The last argument is optional and defaults to SECONDS.

A duration value of 0 will unpin the file.  The command does not allow the user to pin the file indefinitely (the duration value must be a positive integer).

Three new properties associated with the configuration of the pinmanager have been added to nfs.properties:

nfs.service.pinmanager=${dcache.service.pinmanager}
nfs.service.pinmanager.timeout=300000
(one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)nfs.service.pinmanager.timeout.unit=MILLISECONDS
httpd: revert datanucleus-xml version (pom.xml)

The latest version of the datanucleus-xml library has introduced a regression in involving the XSLT properties.

In particular, at org.datanucleus.store.xml.ConnectionFactoryImpl$ManagedConnectionImpl.close calls the release() method:

        public void release()
        {
            if (commitOnRelease)
            {
                try
                {
                    TransformerFactory tf = TransformerFactory.newInstance();
                    tf.setAttribute("indent-number", indent);

However, "indent-number" is not supported; OutputKeys.INDENT should be used
(see http://stackoverflow.com/questions/15134861/java-lang-illegalargumentexception-not-supported-indent-number).

Until the datanucleus team fixes this, we need to revert to the previous version (3.2.0-release, which is used by branches previous to 2.8).

TESTING:

Before patch, refreshing the Alarms table in webadmin produces:

05 Feb 2014 12:11:57 (httpd) [] Exception closing connection to XML file
java.lang.IllegalArgumentException: Unknown attribute indent-number
        at net.sf.saxon.Configuration.setConfigurationProperty(Configuration.java:2185) ~[saxon-8.7.jar:na]
        at net.sf.saxon.TransformerFactoryImpl.setAttribute(TransformerFactoryImpl.java:342) ~[saxon-8.7.jar:na]
        at org.datanucleus.store.xml.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:194) ~[datanucleus-xml-3.2.1.jar:na]
        at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionCommitted(ConnectionManagerImpl.java:413) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.TransactionImpl.internalPostCommit(TransactionImpl.java:554) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:335) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) [datanucleus-api-jdo-3.2.7.jar:na]
        at org.dcache.webadmin.model.dataaccess.impl.DataNucleusAlarmStore.get(DataNucleusAlarmStore.java:181) [classes/:na]

The xml database remains inaccessible.

After patch, page refresh works normally and displays the entries from the xml database.

Target: trunk
Request: 2.8
Patch: http://rb.dcache.org/r/6540/
Require-book: no
Require-notes: no
Acked-by: Gerd
httpd: revert datanucleus-xml version (pom.xml)

The latest version of the datanucleus-xml library has introduced a regression in involving the XSLT properties.

In particular, at org.datanucleus.store.xml.ConnectionFactoryImpl$ManagedConnectionImpl.close calls the release() method:

        public void release()
        {
            if (commitOnRelease)
            {
                try
                {
                    TransformerFactory tf = TransformerFactory.newInstance();
                    tf.setAttribute("indent-number", indent);

However, "indent-number" is not supported; OutputKeys.INDENT should be used
(see http://stackoverflow.com/questions/15134861/java-lang-illegalargumentexception-not-supported-indent-number).

Until the datanucleus team fixes this, we need to revert to the previous version (3.2.0-release, which is used by branches previous to 2.8).

TESTING:

Before patch, refreshing the Alarms table in webadmin produces:

05 Feb 2014 12:11:57 (httpd) [] Exception closing connection to XML file
java.lang.IllegalArgumentException: Unknown attribute indent-number
        at net.sf.saxon.Configuration.setConfigurationProperty(Configuration.java:2185) ~[saxon-8.7.jar:na]
        at net.sf.saxon.TransformerFactoryImpl.setAttribute(TransformerFactoryImpl.java:342) ~[saxon-8.7.jar:na]
        at org.datanucleus.store.xml.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:194) ~[datanucleus-xml-3.2.1.jar:na]
        at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionCommitted(ConnectionManagerImpl.java:413) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.TransactionImpl.internalPostCommit(TransactionImpl.java:554) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:335) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) [datanucleus-api-jdo-3.2.7.jar:na]
        at org.dcache.webadmin.model.dataaccess.impl.DataNucleusAlarmStore.get(DataNucleusAlarmStore.java:181) [classes/:na]

The xml database remains inaccessible.

After patch, page refresh works normally and displays the entries from the xml database.

Target: trunk
Request: 2.8
Patch: http://rb.dcache.org/r/6540/
Require-book: no
Require-notes: no
Acked-by: Gerd
(cherry picked from commit 3dee98213f75ebd100e81c7ee9e129b7decd2b73)

Signed-off-by: alrossi <arossi@otfrid.fnal.gov>
libs: update nfs code to 0.7.2

The libdcap can distinct between regular file systems
and dCache's NFS. This is achieved by existence of a
magic file '.(get)(cursor)', which can be accessed in
any directory in dCache's namespace. Nevertheless, this
behaviour prevents dcap-aware clients to access dcache
through NFSv4.1.

The updated version of nfs library adds a possibility to
hide magic file from dcap client and force them to treat
dCache's NFS as a regular file system.

New export options added: dcap(defualt) and no_dcap.

Example:

/pnfs 10.0.1.1/24(rw,acl,no_dcap)

Changelog for nfs4j-0.7.1..nfs4j-0.7.2
    * [5c4808c] vfs: add a special option to hide dcache from dcap clients

Acked-by: Gerd Behrmann
Target: master, 2.7, 2.8
Require-book: yes
Require-notes: yes