dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape Transfers page)

The Wicket PropertyModel requires either an object field name or a getter/setter
name which can be used to invoke by reflection in order to store and retrieve the property value.

In the case of a ListView, the PropertyModel value is a list; in the case of the TapeTransfersQueue page,
this list is provided by a service whose DAO abstraction accesses a page cache that is refreshed
periodically by calling "xrc ls" on the PoolManager.  The current implementation uses a getter
which extracts a new copy of the list from cache.

This getter, however, is invoked each time the "populateItem" method is called (that is, for each list item);
aside from being inefficient, this could cause an indexing problem if the size of the underlying
list changes in the middle of the procedure to populate the entire list; specifically, the
reported bug suggests that either the original or the current list size is used to determine
the successive iteration, but on that successive call, the list size has been reduced (by one),
causing an IndexOutOfBounds exception.

The solution here is to associate a stable instance of the list with the page constructor.
There is no problem in terms of updating the page values, since this particular page is only
refreshed manually by reloading. A fresh copy of the list will be obtained from the cache on
each reload, which calls the page constructor.

Testing:

Before this patch, loading the page causes the browser to hang and
the following stack trace is observed in the log:

1.04.37 [qtp1040303644-45] [] Unexpected error occurred
org.apache.wicket.WicketRuntimeException: Error attaching this container for
rendering: [Page class = org.dcache.webadmin.view.pages.tapetransferqueue.TapeTransferQueue, id = 0, render count = 1]
...
Caused by: java.lang.IndexOutOfBoundsException: Index: 9739, Size: 9739
at java.util.ArrayList.rangeCheck(ArrayList.java:604) ~[na:1.7.0_25]
at java.util.ArrayList.get(ArrayList.java:382) ~[na:1.7.0_25]
at org.apache.wicket.markup.html.list.ListItemModel.getObject(ListItemModel.java:58) ~[wicket-core-1.5.10.jar:1.5.10]

After patch application, the error no longer occurs.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6527
Require-book: no
Require-notes: yes
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8225
Acked-by: Tigran

RELEASE NOTES:  Fixes a bug in the TapeTransferQueue page whereby an IndexOutOfBounds error
can occur (and hang the page).
(cherry picked from commit ed919cf0eb99da85d594ebbc4d27fba6b1c9d09e)

Signed-off-by: alrossi <arossi@otfrid.fnal.gov>
chimera: Optimized creation time column changeset

After extensive tests, I found a way to significantly improve the time it takes
to add the creation time column.

The key problem is not adding the column but to update every row in the
t_inodes table to populate the new column. Due to the MVCC nature of postgresql
tables, updating a row doesn't replace it - instead a new copy of the row is
added while the existing row is kept until the transaction within which the
update executes is committed. Since every row in the entire table is updated
in a single transaction, the table effectively doubles in size during this
process. In general, when a row is updated, any index pointing to the row needs
to be updated too. This was the source of the high amount of I/O observed on
our postgresql instance.

Postgresql has a trick to reduce the cost of updates: Heap-only tuples (HOT)
allow the updated row to be stored within the same logical block as the
existing row. In such cases the index doesn't need to be updated, as the index
really points to blocks, not individual rows. HOT updates however require that
there is enough free space in blocks to accomodate additional copies of rows.
During regular operation, deleted rows leave such free space. Also updated rows
once the transaction is committed leave the old copy free to be reclaimed.

In this particular update we however update every record in the entire table
within a single transaction. This means that to avoid expensive index updates,
we need enough free space in blocks to duplicate every row. This is exactly
what this patch achieves: It changes the fillfactor (percentage of a block
filled with regular inserts, leaving the rest available for HOT updates) to
45%, leaving plenty of room for adding an additional copy of every row. After
the update the fillfactor is raised to 75% to avoid wasting too much disk space
in the future.  The default fillfactor in postgresql is 100%, which is fine for
mostly static data, but not fine if rows are updated frequently (we update
inodes frequently due to atime updates and due to link count changes for
directories). The addition of the new column is altered too: We now add the
non-null constraint with a computed default value right away, forcing a table
rewrite (which will use the new fill factor). Once the transaction is
committed, old copies of rows can be reallocated, so there will again be 55%
free space in the table.  Note that after raising the fillfactor to 75%, we do
*not* rewite the table to reclaim the space. Postgresql will use the space when
creating new inodes, but an admin can manually reclaim the space by running
vacuum full on t_inodes (only recommended with Postgresql 9.0 or higher - use
cluster on older versions). Unless tight on disk space, this should not be
necessary.

The difference of this change on our test system is tremendous (similar hardware
to our production system, except significantly less RAM). The old changeset took
approximately 41 hours to complete (for around 48 million inodes - the same
changeset took approx 5 hours on our production hardware). The updated changesets
complete in approx. 40 minutes (half of that is for adding the column; the other
half for update the creation time).

The changeset has been written such that it will not run if the column was already
added to t_inodes (as is the case at NDGF).

Target: trunk
Request: 2.8
Require-notes: no
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6530/
(cherry picked from commit b25f8dfbc214d09ae62be3da0a8f16f1c283c612)
Merge pull request #395 from gbehrmann/fix/2.8/rb6530

chimera: Optimized creation time column changeset
ftp: Reenable support for RFC 2428

To avoid that IPv6 enabled clients use extended passive and extended port
commands on IPv4 connections, support for these commands is limited to
IPv6 connections. This is necessary as using these commands prevents
ftp data connections from being redirected to pools.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/r/6525/
Merge pull request #396 from paulmillar/fix/ipv6-ftp

ftp: Reenable support for RFC 2428
Merge pull request #392 from alrossi/fix/2.8/tape-transfer-queue

dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape T...
httpd: revert datanucleus-xml version (pom.xml)

The latest version of the datanucleus-xml library has introduced a regression in involving the XSLT properties.

In particular, at org.datanucleus.store.xml.ConnectionFactoryImpl$ManagedConnectionImpl.close calls the release() method:

        public void release()
        {
            if (commitOnRelease)
            {
                try
                {
                    TransformerFactory tf = TransformerFactory.newInstance();
                    tf.setAttribute("indent-number", indent);

However, "indent-number" is not supported; OutputKeys.INDENT should be used
(see http://stackoverflow.com/questions/15134861/java-lang-illegalargumentexception-not-supported-indent-number).

Until the datanucleus team fixes this, we need to revert to the previous version (3.2.0-release, which is used by branches previous to 2.8).

TESTING:

Before patch, refreshing the Alarms table in webadmin produces:

05 Feb 2014 12:11:57 (httpd) [] Exception closing connection to XML file
java.lang.IllegalArgumentException: Unknown attribute indent-number
        at net.sf.saxon.Configuration.setConfigurationProperty(Configuration.java:2185) ~[saxon-8.7.jar:na]
        at net.sf.saxon.TransformerFactoryImpl.setAttribute(TransformerFactoryImpl.java:342) ~[saxon-8.7.jar:na]
        at org.datanucleus.store.xml.ConnectionFactoryImpl$ManagedConnectionImpl.close(ConnectionFactoryImpl.java:194) ~[datanucleus-xml-3.2.1.jar:na]
        at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionCommitted(ConnectionManagerImpl.java:413) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.TransactionImpl.internalPostCommit(TransactionImpl.java:554) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:335) [datanucleus-core-3.2.11.jar:na]
        at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) [datanucleus-api-jdo-3.2.7.jar:na]
        at org.dcache.webadmin.model.dataaccess.impl.DataNucleusAlarmStore.get(DataNucleusAlarmStore.java:181) [classes/:na]

The xml database remains inaccessible.

After patch, page refresh works normally and displays the entries from the xml database.

Target: trunk
Request: 2.8
Patch: http://rb.dcache.org/r/6540/
Require-book: no
Require-notes: no
Acked-by: Gerd
(cherry picked from commit 3dee98213f75ebd100e81c7ee9e129b7decd2b73)

Signed-off-by: alrossi <arossi@otfrid.fnal.gov>
dcache-core, alarms: remove postgresql precondition on schema

There is nothing in the liquibase schema for alarms which specifically requires postgresql.
Removing this precondition allows the rdbms option to be tested in system-test using hsqldb.

Target: master
Request: 2.8
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/6548/
Require-book: no
Require-notes: no
Acked-by: Gerd
(cherry picked from commit b84b3fa2fe66b44a9f24dcdeda8cc344780f492b)

Signed-off-by: alrossi <arossi@otfrid.fnal.gov>
libs: update to bugfix nfs4j-0.7.3

Changelog for nfs4j-0.7.2..nfs4j-0.7.3
    * [f2b0d2b] vfs: fix regression introduced in 1c916c5

Acked-by: Gerd Behrmann
Target: master, 2.8, 2.7
Require-book: no
Require-notes: no
(cherry picked from commit 2d14212ba231232090277b767b7e3352d60474d1)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Merge pull request #399 from alrossi/fix/2.8/revert-dn-xml-version

httpd: revert datanucleus-xml version (pom.xml)
Merge pull request #402 from alrossi/fix/2.8/alarms-schema

dcache-core, alarms: remove postgresql precondition on schema
Merge pull request #406 from kofemann/nfs-0.7.3-2.8

libs: update to bugfix nfs4j-0.7.3
[maven-release-plugin] prepare release 2.8.0
[maven-release-plugin] prepare for next development iteration
core, billing: remove unit test

The unit test for billing persistence uses timeouts/sleep to check the presence and absence of entries in the database.  This timing, however, is inherently unstable because it depends on the OS and the thread scheduler in particular.  Thus the test will sometimes fail during the build process.

However, the usefulness of this test is very limited, so I propose simply to eliminate it.

Target: 2.8
Patch: http://rb.dcache.org/r/6556/
Require-book: no
Require-notes: no
Acked-by: Gerd
Acked-by: Paul
xrootd: Upgrade to xrootd4j v1.2.4

Contains the following changes over v1.2.3:

42101c6 xrootd4j: Make errors null terminated
5ae3a44 Fix read beyond end of file

These changes improve compatibility with the "new xrootd client".

Target: trunk
Request: 2.8
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8156
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/6557/
(cherry picked from commit 68055d49528c12e714928c0d83418798fb8cf793)
pool: Fix access time update

The bug causes access time update to happen on pool to pool and not on regular
downloads - the reverse is supposed to happen.

This affects sweeper garbage collection order, space cost calculation, and
access time update in the name space.

Target: trunk
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8234
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6564/
(cherry picked from commit 05cb3fcab635d741210801efbcc39bd2590dead1)
Merge pull request #407 from alrossi/fix/2.8/remove-billing-test

core, billing: remove unit test
Merge pull request #411 from gbehrmann/fix/2.8/rb6557

xrootd: Upgrade to xrootd4j v1.2.4
Merge pull request #414 from gbehrmann/fix/2.8/rb6564

pool: Fix access time update