rpm: include logrotate config file

Acked-by: Gerd Behrmann
Target: master, 2.10
Require-book: no
Require-notes: no
billing: Expose session identifier

In particular for SRM transfers, this allows billing entries to be bound to the
srm session.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7354/
pnfsmanager,doors: Add user relative upload directories

This patch adds user relative upload directories. If dcache.user-directory
is defined to a relative path, it is now interpreted relative to the
account's root directory. This works around the problem that there may
not be a common directory accessible through all accounts. Rather than
adding extra FTP doors, this patch allows extra upload directories to
be created.

The downside is that the upload directories may clash with other items
in the users directory. Also, the user could bypass the access limitations
(if he got write permission to the parent of the upload directory).
Finally, the sysadmin has several directories to check for orphaned
temporary upload directories.

All doors and pnfsmanager needs to be updated for the change to be effective.
If the feature is not used, then there is no limitation on upgrades.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7328/
webadmin: minor improvements to rrd4j-based pool-queue plots

There seems to be a bug in the rrd4j code. I have
reported this but am not confident anything will be done about it, since
version 2.2 came out in June of 2013 and no activity is evident
since then. In reality, the bug has already been discovered,
but the original reporter did not follow up by providing
a concrete case or data for verification. This is what I provided.

https://code.google.com/p/rrd4j/issues/detail?id=44

In essence, it seems that the stacked graphs are subject to
ghost artifacts which create a different colored border around
the area; if the time steps are narrow enough, this actually
can look as though it were displaying non-zero values for
a different data set. See the attached for comparison of
24, 48 and 96-hour graphs of the same data.

In the course of investigating this problem, I did discover
that the code was allowing unfeasible mappings of minimum
timesteps to fractions of a pixel. This has now been fixed
by automatically adjusting the width; the code also forces
at least two pixels per timestep (cfr. the 96-hour graph)
by expanding the width. This at least guarantees a certain
amount of legibility and mitigates the border-bug.

A number of other minor improvements and adjustments were
made:

-    The histogram has been slightly rewritten for efficiency;
-    An interface has been added to facilitate (1).
-    Use of the source name derived from the enum has been regularized.
-    "!" has been appended to this name, to enforce, according to rrd4j
    	convention, the conversion of NaNs to 0.0s.
-    The separate refresh interval has been eliminated. The sampling should
    	be congruent with the minimum time step.
-    Local variables have been added for re-used values.
-    The plot display names were being set to ALL CAPS for aesthetics/readability;
    	this has been removed to favor easy cut & paste into the filter
    	from other sources such as the logs or directory listings.
-    Default values for time span have been changed to something more reasonable
    	(48 hours rather than two weeks).

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Karsten
Require-book: no
Require-notes: yes

NOTE: The refresh property need not be removed in the backports (it will
just be ingored by the code).

Testing: Redeployed and observed plotting behavior (see attached) on testbed.

RELEASE NOTES:

Small adjustments to the pool queue plots in order to improve
appearance.  Some property defaults have been changed.  Note:
this patch does not entirely remedy what appears to be a bug
in rrd4j (see https://code.google.com/p/rrd4j/issues/detail?id=44).
billing: Add billing format for PoolHitInfoMessage

Otherwise the Indexer is unable to parse these billing entries and then
appear as empty records in the json and yaml output.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7355/
billing: Reduce startup time of dcache billing command

The patch also moves the class path utility functions to loadConfig as
they are not related to dcache domains/services.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7356/
billing: Improve error message on failur to parse date arguments

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7359/
billing: Add file size to request records

Request records submitted by doors lack the file size (they always
appeared as 0 in billing records). This patch adds the file size for
all protocols except for NFS.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7360/
billing: Fix date handling for json and yaml output

The raw date format in billing files lacks a year. The billing utility
has worked around that by prefixing the output with the full date. For
JSON and YAML output, this leads to a faulty timestamps like this:

    Jan 1, 2200: 01.01 00:00:00

This patch resolves the problem by converting it to an ISO8601 compliant
format (the preferred format for JSON and YAML).

The raw output is also adjusted such that the default format is prefixed
by the year rather than the full date.

JSON and YAML output formats have beeen adjusted to skip unrecognized
records. For YAML, the raw billing line is included as a comment.

Finally, the patch fixes an issue with outputing JSON from multiple days.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7357/
nfs4:  shutdown state handler on destroy

for graceful dcache shutdown

Target: master, 2.10
Acked-By: Paul Millar
Require-book: no
Require-notes: no
pool: ensure that mover error message always set

replace null objects with empty strings. update
all calls to iAbstractMover#setTransferStatus to
set non null message.

Acked-by: Paul Millar
Acked-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no
chimera: query for inode generation on readdir

keep nfs client cache more consistent.
add missing rdev field.

Acked-by: Paul Millar
Target: master, 2.10
Require-book: no
Require-notes: no
admin: Propopage NPE to user

I ran into a bug in migration module that caused an NPE to be thrown. Unfortunately
the admin shell return no message at all. This was because the exception was wrapped
in a CacheException with an empty message string.

This patch addresses this problem at several levels: Unexpected exceptions are
now wrapped with message being Throwable#toString (given that we don't know which
except is being wrapped it is reasonable to include the exception type in the error
message), RuntimeExceptions and Errors are propagated unwrapped, and the log output
in admin is increased from warning to error.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7361/
pool: Fix pool selection bugs in migration module

There were two bugs in migration module:

- Random pool selection would select pools even when they are full.

- Proportional pool selection would trigger a NPE as it would return
  null when all pools are full.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7362/
cell: Log exceptions within the correct cell context

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7363/
pool: Make migration task cancellation more robust

A recent bug as a side effect caused task cancellation to fail
because cancelCopy threw an NPE. As a result the task got stuck
in this state. This patch moves the timeout initiation to be the
first action when entering the Cancelling state. That way the task
would eventually have timed out.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7364/
webadmin: fix exit login in billing refresh loop

Currently the thread which refreshes the billing plots
will exit if it encounters an unspecified error from
the billing service. But this behavior does not
take into account slow start-up of domains (i.e.,
the billing service may not be there yet but will
be eventually).

The patch fixes the logic to treat NoRouteToCell
exceptions differently by waiting for a short time
and retrying.

Testing done: On deployed service without billing,
and then with billing booted.

Target: master
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Patch: https://rb.dcache.org/r/7306
Acked-by: Gerd
Require-note: yes
Require-book: no

RELEASE NOTES:
Fixes a bug where a simple timeout (NoRouteToCell) causes an
exit from the billing service refresh loop requiring a restart
of the domain in order to reconnect.
webadmin: make pool queue data enum Java 8 compatible

The static method seems to cause problems for java 8 compilation.
Evidently there is something different about the availability of
of the enum instance reference inside the class definition.

But the better solution is just a shared instance method anyway.
Is there a need to change this in the backports? I imagine
systems won't be updating to java 8 unless installing 2.12 ...

Testing:

This compilation error goes away with the patch:

[ERROR] /Developer/Work/fnal/dCache/dcache/modules/dcache-webadmin/src/main/java/org/dcache/webadmin/model/businessobjects/rrd4j/PoolQueuePlotData.java:[90,24] no suitable method found for getSourceName(<anonymous org.dcache.webadmin.model.businessobjects.rrd4j.PoolQueuePlotData.PoolQueueHistogram>)
method org.dcache.webadmin.model.businessobjects.rrd4j.RoundRobinDatabaseDatasource.getSourceName() is not applicable
(actual and formal argument lists differ in length)
method <anonymous org.dcache.webadmin.model.businessobjects.rrd4j.PoolQueuePlotData$PoolQueueHistogram$1>.getSourceName() is not applicable
(actual and formal argument lists differ in length)

Target: master
Patch: https://rb.dcache.org/r/7370
Acked-by: Gerd
Require-book: no
Require-notes: no

Note that the pull requests for rb 7346 have been updated to take care of this.
pool: Fix ISE in 'rep ls'

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7369/
pool: Fix ISE in CacheEntryImpl#toString

Fixes

18 Sep 2014 12:57:38 (pdc_kth_se_034) [admin] Uncaught exception in thread sweeper-free
java.lang.IllegalStateException: Attribute is not defined: STORAGEINFO
        at org.dcache.vehicles.FileAttributes.guard(FileAttributes.java:150) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.vehicles.FileAttributes.getStorageInfo(FileAttributes.java:296) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.repository.v5.CacheEntryImpl.toString(CacheEntryImpl.java:139) ~[dcache-core-2.8.4.jar:2.8.4]
        at java.lang.String.valueOf(String.java:2854) ~[na:1.7.0_51]
        at java.lang.StringBuilder.append(StringBuilder.java:128) ~[na:1.7.0_51]
        at org.dcache.pool.classic.SpaceSweeper2.reclaim(SpaceSweeper2.java:335) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.classic.SpaceSweeper2.access$000(SpaceSweeper2.java:39) ~[dcache-core-2.8.4.jar:2.8.4]
        at org.dcache.pool.classic.SpaceSweeper2$1.run(SpaceSweeper2.java:209) ~[dcache-core-2.8.4.jar:2.8.4]
Waiting for data... (interrupt to abort)

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7368/