srm: Respect read-only session in 2.2 (equivalent of http://rb.dcache.org/r/5633/ for java 6 based 2.2)

This patch fixes the issues by attempting to failfast - fail SRM request even before the request handler is invoked and
there fore before requests are scheduled (in case of srmput). It is achieved by adding isReadOnly() method to SRMUser interface.

Ticket: 7842
Acked-by: Al Rossi
Target: 2.2
Bug: http://rt.dcache.org/Ticket/Display.html?id=7842
Request: 1.9.12
Require-book: no
Require-notes: yes

RELEASE NOTES
SRM respects read-only session qualifier (as specified in storage-authzb or dcache.kpwd). That is
if login record contains "read-only" qualifier SRM will refuse performing namespace modification
functions with SRM_AUTHORIZATION_FAILURE error code
poolmanager, doors: Don't retry stage requests on the same pool

Addresses a regression introduced between 1.9.12 and 2.2. The regression
means that poolmanager would not remember the previously tried stage pool
and thus could select the same pool twice in a row when retrying failed
stage requests.

The patches addresses this problem. To be effective, both pool manager
and doors have to be updated, although the patch does not break any
version compatibility.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5678/
(cherry picked from commit 4174110f6d6083908a8937e42e1c4969596e53ac)

Conflicts:
	modules/dcache/src/main/java/diskCacheV111/poolManager/RequestContainerV5.java
cells: Fix race conditions in cell lookup

When cells are killed, they are moved from one map to another map. This operation
is not atomic and methods that look in both places need to be synchronized.

This addresses a null pointer exception experienced in the DCAP door:

java.lang.NullPointerException: null
        at diskCacheV111.doors.DCapDoorInterpreterV3.ac_get_door_info(DCapDoorInterpreterV3.java:733) ~[dcache-dcap-2.2.13.jar:2.2.13]
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_19]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_19]
        at dmg.util.CommandInterpreter.execute(CommandInterpreter.java:609) ~[cells-2.2.13.jar:2.2.13]
        at dmg.util.CommandInterpreter.command(CommandInterpreter.java:486) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellAdapter.executeLocalCommand(CellAdapter.java:1020) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:940) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1051) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:949) ~[cells-2.2.13.jar:2.2.13]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_19]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_19]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:654) [cells-2.2.13.jar:2.2.13]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_19]

Target: trunk
Request: 2.6
Request: 2.2
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7896
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5700/
(cherry picked from commit 45850f5d50d867772d91b8526d9b7210ae9aaa55)

Conflicts:
	modules/cells/src/main/java/dmg/cells/nucleus/CellGlue.java
cells: Fix deadlock

Addresses a regression introduced in 45850f5d50d867772d91b8526d9b7210ae9aaa55. The
regression introduces a deadlock caused by low-level class loading logic in AspectJ
using the logging system, and our logging system looks up cells through getCell(),
a method made synchronized by the above commit.

The solution adopted is to not have two cell maps. Rather we have a set of killed
cells to track the fact that a cell was killed.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5718/
(cherry picked from commit 0d67d01d9fc3a33a4d5dbe26019d959accad325f)

Conflicts:
	modules/cells/src/main/java/dmg/cells/nucleus/CellGlue.java
	modules/cells/src/main/java/dmg/cells/nucleus/CellShell.java
all: Update voms-api to 2.0.6

We are using an unsupported and buggy voms-api version
Switch to voms-api to 2.0.6 fixing "Certificate verification: Maximum certification path length exceeded." exception problem

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7848
Acked-by: Gerd
Target: 2.2
Require-book: no
Require-notes: yes
(2.2) dcache (billing) improve error handling in plotting classes

module: dcache (billing plots)

In version 2.2, the plotting code is still from the original version of billing plots (2011), and thus has a number of places which were catching Throwable, or where duplicate logging was implemented.  I had promised to fix these but never got around to it.

This patch changes the Throwable to Exception or something more specific where possible, and eliminates the duplicate logging.  It also eliminates the stack trace at the error level.

Note that DataNucleus unfortunately will continue to log the entire stack trace for errors at the query level.  This is seen from the code, for instance, to

 protected Object performExecuteTask(final Object... args) {
  ...
        catch (ExecutionException e)
        {
            // Exception thrown executing the query
            tasks.remove(Thread.currentThread());
            NucleusLogger.QUERY.error(LOCALISER.msg("021021", toString()), e);
            throw (NucleusException)e.getCause();
        }
 }

where 021021 is the externalized local message "An error occurred when executing query {0}".  This is what we observe still in our logs when there is an SQL error.

I have also decided it makes sense to exit from the plotting loop in this case.

Some of this behavior was modified when I rewrote this code for 2.6/2.7.  I will nevertheless still check that the error handling conforms there with the current changes.

Testing done:

After application of patches, noise level in log on SQL errors is greatly reduced, and plotting is shut off on an error.

Target: 2.2
Patch: http://rb.dcache.org/r/5666
Require-notes: yes
Require-book: no
Acked-by: Gerd
Merge-req: https://github.com/dCache/dcache/pull/107

RELEASE NOTES:

The log for the domain where billing plots are generated has been made less verbose at the error level.  If there is a critical/severe (i.e., SQL) error encountered, plotting will stop and will only restart if the domain is restarted.
(skel): time-unit corrections to billing migration scripts

The scripts contained in skel/share/migration which are intended to aid in migrating preexisting aggregated billing data or in aggregating the data from the main fine-grained tables have two errors in them.

1.  migrate_from_preexistent.sql does not account for the time-unit differences between the original dc_tm_daily tables, which store the data in seconds, and the new tables, which preserve the millisecond unit of the base tab$

2.  migrate_from_messageinfo.sql neglected to exclude data from failed operations (errorcode = 0).

This patch repairs these two issues.

Testing	done:

Before the patches, stored and displayed data did not correspond with previous installations, nor did the two scripts produce similar results.  After the patch, the results of both migrations are consonant with prior data.

Target: master@ded9bf4766c55e3a2f81f8dc25641a5921030c4c
Request: 2.6
Request: 2.2
Patch: http://rb.dcache.org/r/5732/
Require-book: no
Require-notes: yes
Acked-by: Gerd

RELEASE NOTES:

Fixes two bugs in the migration scripts.  One which neglected to convert from seconds to milliseconds (migrate_from_preexistent.sql) for connection time values, the other which neglected to exclude data from failed operations$

Note for those who already have applied the faulty script: the migration from the preexistent tables does not really affect anyone outside of Fermi.  The other migration error generates differences which are probably negligib$
Merge pull request #121 from gbehrmann/fix/2.2/rb5700

cells: Fix race conditions in cell lookup
Merge pull request #96 from DmitryLitvintsev/fix/2.2/readonlysrm

srm: Respect read-only session in 2.2 (equivalent of http://rb.dcache.or...
Merge pull request #115 from gbehrmann/fix/2.2/rb/5678

poolmanager, doors: Don't retry stage requests on the same pool
Merge pull request #124 from chrber/fix/2.2/rb5712

all: Update voms-api to 2.0.6
cells: Fix retry loop and logging in sendAndWaitToPermanent

Signed-off-by: Gerd Behrmann
merge: trunk@18210
Update to Spring Framework 3.1.4

We have had reports about the following problem in pin manager:

01 Mar 2013 14:08:27 (PinManager) [] Unexpected failure while unpinning
java.lang.IllegalStateException: No JDO PersistenceManager bound to thread, and configuration does not allow creation of non-transactional one here
        at org.springframework.orm.jdo.PersistenceManagerFactoryUtils.doGetPersistenceManager(PersistenceManagerFactoryUtils.java:153) ~[spring-orm-3.1.0.RELEASE.jar:3.1.0.RELEASE]
        at org.springframework.orm.jdo.TransactionAwarePersistenceManagerFactoryProxy$PersistenceManagerFactoryInvocationHandler.invoke(TransactionAwarePersistenceManagerFactoryProxy.java:159) ~[spring-orm-3.1.0.RELEASE.jar:3.1.0.RELEASE]
        at $Proxy11.getPersistenceManager(Unknown Source) ~[na:na]
        at org.dcache.pinmanager.JdoDao.all(JdoDao.java:188) ~[dcache-core-2.2.8.jar:2.2.8]
        at org.dcache.pinmanager.UnpinProcessor.unpin(UnpinProcessor.java:67) ~[dcache-core-2.2.8.jar:2.2.8]
        at org.dcache.pinmanager.UnpinProcessor.run(UnpinProcessor.java:49) ~[dcache-core-2.2.8.jar:2.2.8]
        at org.dcache.util.FireAndForgetTask.run(FireAndForgetTask.java:31) [dcache-core-2.2.8.jar:2.2.8]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [na:1.6.0_31]
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [na:1.6.0_31]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [na:1.6.0_31]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [na:1.6.0_31]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [na:1.6.0_31]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [na:1.6.0_31]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_31]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_31]
        at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]

It would seem that in some cases the @Transactional annotation of the
DAO is not respected. I have observed this myself after restarting
pinmanager. Another restart usually solves the problem.

I found the following bug report at Spring Source:
https://jira.springsource.org/browse/SPR-9857 The problem being reported
is in a completely different context, but seems to be about the load
order of when aspects are woven. It's a long shot, but maybe the fix
cures our problem too. Hence this patch upgrades Spring from 3.1.0 to
3.1.4 in dCache 2.2.

Target: 2.2
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7694
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7683
Merge-req: http://rt.dcache.org/Ticket/Display.html?id=7710
Signed-off-by: Gerd Behrmann
poolmanager: Fixes rebalancer bug related to use of non-US locale

Rebalancer works by issuing migration commands to pools of a
pool group. The code uses String.format to build the migration
command. The formatting of floating point values is sensitive
to the default locale. For a non-US local this means that
floating points will use a comma as a decimal separator. This
will fail with the migration module.

This patch fixes the problem.

Target: trunk
Request: 2.5
Request: 2.2
Request: 1.9.12
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5274/
Signed-off-by: Gerd Behrmann <behrmann@nordu.net>
Merge-request: http://rt.dcache.org/Ticket/Display.html?id=7705

Conflicts:
	modules/dcache/src/main/java/org/dcache/poolmanager/Rebalancer.java
(2.2) statistics: Fix NPE in Html tree creation

It has been noted that the statistics module fails to produce the top level html and to run the collection cron correctly.  In the first case, one sees an error such as the first reported below.  In the second, a different er$

As the code currently stands, the "count" array used in several methods remains uninitialized until inside the subsequent loop.  However, initialization is not loop-invariant, as those loops may fail to run under certain cond$

This patch provides the appropriate pre-loop initialization of the array.

Note that this class has a number of dead variables and methods.  I did not eliminate them, however, as it occurs to me that one might wish to do a bit more cleanup and "modernize" this code.  I leave this to another patch.

Before this patch, one observes the following.

1.  Using the admin command create html, one sees:

Exception in creating html tree for : /var/lib/dcache/billing/2013/06/05/2013-06-05-13.raw
java.lang.NullPointerException: null
at diskCacheV111.services.PoolStatisticsV0.printTotal(PoolStatisticsV0.java:1204) ~[dcache-core-2.6.1.jar:2.6.1]
at diskCacheV111.services.PoolStatisticsV0.prepareDailyHtml(PoolStatisticsV0.java:1136) ~[dcache-core-2.6.1.jar:2.6.1]
at diskCacheV111.services.PoolStatisticsV0.prepareDailyHtmlFiles(PoolStatisticsV0.java:1041) ~[dcache-core-2.6.1.jar:2.6.1]
at diskCacheV111.services.PoolStatisticsV0.access$1100(PoolStatisticsV0.java:116) ~[dcache-core-2.6.1.jar:2.6.1]
at diskCacheV111.services.PoolStatisticsV0$HourlyRunner.run(PoolStatisticsV0.java:432) ~[dcache-core-2.6.1.jar:2.6.1]
at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.6.1.jar:2.6.1]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]

2.  Using the admin command create stat, one sees:

at diskCacheV111.services.PoolStatisticsV0.getPoolRepositoryStatistics(PoolStatisticsV0.java:1361) ~[dcache-core-2.6.6-SNAPSHOT.jar:2.6.6-SNAPSHOT]
at diskCacheV111.services.PoolStatisticsV0.createStatisticsMap(PoolStatisticsV0.java:838) ~[dcache-core-2.6.6-SNAPSHOT.jar:2.6.6-SNAPSHOT]
at diskCacheV111.services.PoolStatisticsV0.createHourlyRawFile(PoolStatisticsV0.java:973) ~[dcache-core-2.6.6-SNAPSHOT.jar:2.6.6-SNAPSHOT]
at diskCacheV111.services.PoolStatisticsV0.access$500(PoolStatisticsV0.java:116) ~[dcache-core-2.6.6-SNAPSHOT.jar:2.6.6-SNAPSHOT]
at diskCacheV111.services.PoolStatisticsV0 $HourlyRunner.run(PoolStatisticsV0.java:400) ~[dcache-core-2.6.6-SNAPSHOT.jar:2.6.6-SNAPSHOT]
at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:657) [cells-2.6.6-SNAPSHOT.jar:2.6.6-SNAPSHOT]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0]
15 Jul 2013 15:57:08 (PoolStatistics) [] prepareDailyHtmlFiles : File not found : /var/lib/dcache/statistics/2013/07/15/2013-07-15-day.drw

After the patch is applied, both of these commands work again, and the top index.html is produced.

Target: master@1ab1d24b01a614e8011a3be0915dced3dbb90ee8
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=785
Bug: http://rt.dcache.org/Ticket/Display.html?id=785
Acked-by: Gerd

RELEASE NOTES:

Fixes NullPointer bug which prevents under certain conditions top-level index.html from being created (and the statistics page being found in the browser).
replicamanager: Fix NullPointerException

Fixes an NPE when database connection checkout fails:

08 mar 2013 19:56:57 (replicaManager) [] setHeartBeat() ERROR: Can't add/update process 'Adjuster' status in 'heartbeat' table in DB
java.lang.NullPointerException: null
at diskCacheV111.replicaManager.ReplicaDbV1.setHeartBeat(ReplicaDbV1.java:1160) ~[dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at diskCacheV111.replicaManager.ReplicaManagerV2$Adjuster.run(ReplicaManagerV2.java:917) [dcache-core-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:678) [cells-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_11]

Target: trunk
Request: 2.5
Request: 2.2
Require-notes: yes
Require-book: no
Patch: http://rb.dcache.org/r/5273/
Committed: master@67d8a291795b6492594543dd16660ccddc89899f
Signed-off-by: Gerd Behrmann <behrmann@nordu.net>
Merge pull request #130 from alrossi/fix/2.2/statistics_npe

(2.2) statistics: Fix NPE in Html tree creation
System cell: fix kill command

The patch http://rb.dcache.org/r/4278/, committed to master@7071473dc27df13
(pre-2.2) and merged 1.9.12@51e03907399a510 fixes the problem that a
kill command may take time to complete. In particular, when working
around a configuration issue, an existing gPlazma is killed before a
fresh gPlazma is started. If the new gPlazma is started before the old
cell has died then the new gPlazma will not start. This is the
race-condition the patch fixes.

Unfortunately, the patch also introduces a deadlock. When attempting
to kill the System cell, the message-processing thread is blocked
waiting for a signal that the cell has died. This signal is generated
when processing an event by the message-processing thread which,
because the kill command is blocking, will never happen.

This patch fixes the problem by replying with a DelayedReply; an
implementation of the future Reply. The patch also adds support for
replying with a Reply to SystemCell.

Target: trunk
Request: 2.2
Request: 1.9.12
Require-notes: yes
Require-book: no
Patch: http://rb.dcache.org/r/5115/
Acked-by: Gerd Behrmann
Committed: master@d20662a349d6a520fa1dd3b5dba61d7b905b9d97
Merge pull request #131 from paulmillar/fix/2.2/fix_kill_cmd

System cell: fix kill command
fix issue with checksum calculation. FileChannel has to be set to 0th
position before checksum is calculated

Require-book: no
Require-notes: yes
Bug: http://rt.dcache.org/Ticket/Display.html?id=7579
Acked-by: Tigran Mkrtchyan
Target: trunk
Request: 2.2
Request: 2.4
Request: 2.5