srm: Clean up mkdir and rmdir implementations

Improves error codes and error messages of srmMkdir and srmRmdir
operations. Improves performance of srmRmdir.

By moving some functionality into the Storage class, the logic
could be simplified and made faster.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6120/
srm: Clean up srmRm implementation

Improves return codes and error messages.

The batching logic is changed to not be as bursty as the before.
Rather than submit a patch and wait for all to finish before
submitting the next batch, the code now submits new requests as
previous requests finish. Thus remove requests will be more
interleaved with other PnfsManager requests, and we can maintain
a high throughput with a lower limit than before. For this reason
I reduced the default value of srm.limits.remove-batch-size to 50.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6121/
Define default transaction isolation level for databases

BoneCP doesn't remember the default isolation level of the
actual Connection. Instead it injects a configured default,
or leaves the setting untouched if not defined. The latter
means that the isolation level is unpredictable if it is
changed by some code and then the connection is recycled
by some other code.

In the SRM this leads to errors like:

16 Oct 2013 22:33:37 (SRM-sisyphus) [131.169.72.212:51026 3Lg:9:srm2:prepareToPut] Please report this failure to support@dcache.org
org.springframework.dao.CannotSerializeTransactionException: PreparedStatementCallback; SQL [UPDATE srmrequestcredentials SET creationtime=?, credentialname=?, role=?,  numberofusers=?, delegatedcredentials=?, credentialexpiration=? where id=? ]; ERROR: could not serialize access due to concurrent update; nested exception is org.postgresql.util.PSQLException: ERROR: could not serialize access due to concurrent update
	at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:273) ~[spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]

This error is caused by the isolation level being higher than
expected.

This patch sets the default to READ_COMMITTED. This matches
the PostgreSQL default, which is what our code is developed
for.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6122/
srm: Clean up srmMv implementation

Minor improvements to return codes and error messages.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6123/
srm: Clean up srmReserveSpace and srmReleaseSpace implementations

Addresses correctness issues in return codes and
improves error messages.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6125/
packaging: Obsolete dcache-server to make dcache package directly releasable to EMI, EGI

Obsolete dcache-server to make dcache package directly releasable to EMI, EGI*

Target: master
Request: 2.6
Request: 2.2
Requires-notes: no
Requires-book: no
nfs4: include open stateid into diagnostic printouts

like:

[dcache-lab000.desy.de] (dcache-lab002-A) admin > mover ls
4 : RUNNING : 0000DAF610008D6740EE98CAE1EB03DCFEAE h={NFSv4.1/pNFS,OS={[526654a200000001533ad0dd, seq: 55]},cl={131.169.185.68}} bytes=80740352 time/sec=0 LM=0

[dcache-lab000.desy.de] (nfsv41@nfs) admin > info
--- door (dCache implementation of NFSv4.1 MDS) ---
NFS ([3, 4.1]) door (MDS):
  IO queue:
  Known pools (DS):

    dcache-lab002-A : [DS: 00000002000000000000000000000000, InetAddress: [/131.169.191.142:33136]]

  Known movers (layouts):
    000084866518468F4B2D8F01516B27A4F64C : 1404@dcache-lab002-A, OS={[52664e9500000001f141762e, seq: 8]}

Acked-by: Paul Millar
Target: master, 2.6
Require-book: no
Require-notes: no
dcap: convert ClosedByInterruptException into InterruptException

this prevents pools to disable by JTM or mover kill

Acked-by: Paul Millar
Target: master, 2.6
Require-book: no
Require-notes: no
billing: Minor fixes in billing indexer

Addresses some warnings by Jenkins.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6089/
srm: Clean up query operations

Addresses issues in choosing the correct error code,
producing good error messages, and also fixes a couple
of null pointer exceptions and other errors.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6124/
srm: Minor clean up in transfer protocol related methods

Also fixed a potential race condition by switching to a
Guava based cache.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6127/
Library and plugin updates

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6132/
dcache, webadmin: fix exception handling

This commit covers three patches:

http://rb.dcache.org/r/6022/
http://rb.dcache.org/r/6029/
http://rb.dcache.org/r/6030/

BILLING:

A number of checked exceptions which were wrapping unchecked exceptions have been eliminated.  In general RuntimeExceptions are propagated. Two exceptions:  when generating JAIDA plots, and within the webadmin refresh() call to the billing service.  In the latter case, JVM errors are propagated, but RuntimeExceptions simply cause the the loop to exit.

CORE, XACML, WEBADMIN:

Code in several places is not handling exceptions correctly; in particular, RuntimeExceptions masked by Exception are occasionally being caught and not propagated.

Where it is possible, catch(Exception) has been eliminated; where not possible, catch(RuntimeException) is called first so that this exception can be rethrown.

In several cases, a RuntimeException (such as the JDOExceptions from DataNucleus) are not propagated, as they cannot reasonably be handled elsewhere, and they should not be fatal to the JVM in any case.

A separate set of patches have been reviewed for 2.6.

Testing:

Junit tests rerun.  Deployed and checked.

Target: master
Request: 2.6
Acked-by: Karsten
Acked-by: Tigran
Acked-by: Dmitry
Require-notes: no
Require-book: no
dcache-webadmin: fix page update

This patch addresses several RT tickets stemming from the same problem, one which has already been corrected for the CellServices page (see http://rb.dcache.org/r/5807).

This has to do with the way Wicket PropertyModels were provided for pages which maintain lists which need to be updated through the service layer.  In most of these cases, the list was set in the constructor but was never updated during page/form refresh.  The changes adopted here follow the same general pattern of replacing the field accessed by the model with a getter method which also takes care of calling the service to update the collection in question.

A separate patch will be needed for 2.6.

Testing:

Junit tests rerun.  Redeployed and checked updating by enabling/disabling pools.

RELEASE NOTES:
Fixes an issue with some of the tables in the webadmin service which were not refreshing.  Update correctly occurs now on reload, resubmission of the form through link clicking.

Target: master
Patch: http://rb.dcache.org/r/6080
Require-notes: yes
Require-book: no
Bug: http://rt.dcache.org/Ticket/Display.html?id=7929
Bug: http://rt.dcache.org/Ticket/Display.html?id=7964
Bug: http://rt.dcache.org/Ticket/Display.html?id=7983
Acked-by: Gerd
Acked-by: Tigran
dcache-webadmin: fix table sorting for all relevant pages

This patch addresses both a general and specific problem.  Sorting of tables via javascript was not working universally.  This depended on the following sub-issues:

1.  The change to Wicket 1.6 also changed the order in which script is added to the header (as well as the underlying Java APIs for dealing with it); in its current state, this was producing inconsistent nesting of scripts which the browser could not handle.

2.  The sorting code in javascript depended on css id (#), but if there is more than one table per page, this does not work for the subsequent tables.

3.  Not all relevant tables were tagged correctly.

To address the first issue, some refactoring was done.  BasePage now has a subclass, SortableBasePage.  The base page provides an adapter inner method for rendering the page head markup which should be overridden (always calling super first) with the specific script or references for that subclass.  The page itself should provide all the necessary head material for any of the panels it may use.

The second issue is fixed by altering the css and the javascript to use class instead of id.

The final issue was fixed by adding the necessary "class=sortable" marker to the table.

A separate patch will be necessary for 2.6.

The bug reference below is really more of a feature request which is only partially handled here.

Testing:

Clicking around by hand.

Target: master
Patch: http://rb.dcache.org/r/6083
Bug: http://rt.dcache.org/Ticket/Display.html?id=7938
Require-notes: yes
Require-book: no
Acked-by: Gerd

RELEASE NOTES:
Fixes issues preventing the sorting of tables and subtables in webadmin.
dcache-webadmin: add autorefresh to appropriate pages

Adds Ajax auto refresh behavior to appropriate webadmin pages.

This entails the addition of a few shared protected methods, and the adjusting of markup accordingly.

Testing:

Redeployed and observed changes effected through admin service.

Target: master
Patch: http://rb.dcache.org/r/6085/
Require-notes: yes
Require-book: no
Acked-by: Tigran

RELEASE NOTES
CellServices, PoolQueues, PoolGroups, Active and Tape transfer pages now autorefresh, in addition to Alarms, Billing and PoolQueue plots.
chimera: use ArrayList instead of LinedList

NFS listing is heavy with List#get operations, which
is the weakest point for LinkedList. This change improves
listing of jumbo directories ( ~100k files ) by factor of 10.

RELEASE NOTES:
speedup directory listing of large directories

Acked-by:
Target: master, 2.6
Require-book: no
Require-notes: yes
gplazma: ldap: fix closed connection problem

This patch fixes the problem of CommunicationException due to externally
closed connections (e.g., by a firewall) and it introduces unit tests
for all public methods of the plugin.

Acked-by: Tigran
Target: master
Request: 2.7
Require-book: no
Require-notes: no
srm: Clean up srmPrepareToGet

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6136/
srm: Clean up srmPrepareToPut

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6138/