ftp: Fix root path validation for upload directory

The ftp door can be configured to use a static root rather than the user's
root path. If the static root is not a prefix of the user's root directory,
the door should only allow access to the upload directory. The logic that
enforces this was broken, resulting in such upload attempts failing.

Target: trunk
Request: 2.10
Request: 2.9
Require-notes: yes
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7347/
spacemanager: fix forbidden property message

Patch https://rb.dcache.org/r/7168/ committed as 8ea811d576, marks
deprecated properties as forbidden.  Unfortunately the message for
some spacemanager properties was incorrect and following the advice
leaves sites broken.

This patch updates the comment.

Target: master
Requires-notes: no
Requires-book: no
Patch: https://rb.dcache.org/r/7309
Acked-by: Gerd Behrmann
system-test: add 'start' profile

Add a maven profile option to automatically start (or restart) dCache
as part of the system-test build.

Target: master
Requires-book: false
Requires-notes: false
Patch: https://rb.dcache.org/r/7314/
Acked-by: Karsten Schwank
cell: Declassify eval failure as a bug

Currently, this is claimed to be a bug:

[Gerds-MacBook-Pro.local] (System) admin > eval sdsd
There is a bug here, please report to support@dcache.org: (3) Eval Result : sdsd

After applying this patch, we get:

[Gerds-MacBook-Pro.local] (System) admin > eval sds
(3) Eval Result : sds

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7353/
cell: Prevent interpreter stack overflow from killing the domain

This no longer causes a dCache crash:

cd System
define env x
exec env x
.
exec env x

Now it simply outputs an error message stating that a stack overflow
happened. Before it caused the domain to restart.

Target: trunk
Require-notes: yes
Require-book: no
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7352/
srm: Fix listing of failed, done and cancelled jobs

Both the condition and the ORDER BY clause of the SQL query were
faulty.

The faulty expression can be traced all the way back to 2007 - isn't
that wonderful?

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7350/
pool: Respect LAN port range for internal srmcp transfers

When copying files internally within the same dCache using server side srmCopy,
the actual transfer is done through DCAP. The mover implementing this did not
respect the LAN port range. This patch resolves that problem.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7348/
srm-client: Output file level errors for sync requests

srm clients fail to output file level error information if the initial srmPrepareToPut,
srmPrepareToGet or srmCopy request fails. This would be the common way for synchroneous
process to fail. Originally these requests were always asynchronous and this problem
was not visible to the client.

This patch fixes this for srmCopy, srmPrepareToGet and srmPrepareToPut.

Without the patch one could observe:

Gerds-MacBook-Pro:dcache-git behrmann$ srmcp file:////bin/bash srm://localhost/private/test2
srm copy of at least one file failed or not completed
srm client error:
java.io.IOException: srmPrepareToPut submission failed, unexpected or failed status : SRM_FAILURE explanation= File requests have failed.

with the patch one now gets:

Gerds-MacBook-Pro:dcache-git behrmann$ srmcp file:////bin/bash srm://localhost/private/test2
Fri Oct 10 13:35:55 CEST 2014: retrieval of surl srm://localhost/private/test2 failed, status = SRM_AUTHORIZATION_FAILURE explanation=Access denied: /private/test2
srm copy of at least one file failed or not completed

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7349/
pool: Fix interpretation of eager option of migration module

Migration module will try to reuse existing replicas on target pools. This is
how idempotence is achieved. When a pool with an existing replica fails to
process a request from migration module, the request will patiently be retried.
This can be surpressed using the -eager option. If used, this causes the
migration module to proceed to creating a new replica.

There is a flaw in the interpretation of not specifying -eager: If a pool is
down and not registered in pool manager, the pool information provided by pool
manager does not include those pools. Thus even though existing copies existing
on target pools, those pools are invisible to migration module. Thus despite
not being eager, a migration job would proceed with creating new replicas.

This patch fixes the problem by including offline pools in the query reply from
pool manager. The information is added in such a way that old pools without
this patch will not be affected by the change. If a pool is updated without
updating pool manager, the message will return an empty list for offline pools,
effectively maintaining the old semantics. This to fix the problem, both pool
manager and pools have to be updated, but the change is fully backwards
compatible.

The semantics of the -include-when and -exclude-when options is tricky as they
break idempotence. For offline pools, these expressions cannot be interpreted
and thus a migration job may block on an offline pool that would have been
excluded by these expressions had the pool been online. The proper fix is to
replace the -include-when and -exclude-when options with something that is
idempotent, but such a change would not be applicable to stable branches. The
present patch is kept minimal so it can be included on stable branches.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: https://rb.dcache.org/r/7345
rpm: include logrotate config file

Acked-by: Gerd Behrmann
Target: master, 2.10
Require-book: no
Require-notes: no
billing: Expose session identifier

In particular for SRM transfers, this allows billing entries to be bound to the
srm session.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7354/
pnfsmanager,doors: Add user relative upload directories

This patch adds user relative upload directories. If dcache.user-directory
is defined to a relative path, it is now interpreted relative to the
account's root directory. This works around the problem that there may
not be a common directory accessible through all accounts. Rather than
adding extra FTP doors, this patch allows extra upload directories to
be created.

The downside is that the upload directories may clash with other items
in the users directory. Also, the user could bypass the access limitations
(if he got write permission to the parent of the upload directory).
Finally, the sysadmin has several directories to check for orphaned
temporary upload directories.

All doors and pnfsmanager needs to be updated for the change to be effective.
If the feature is not used, then there is no limitation on upgrades.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: https://rb.dcache.org/r/7328/
webadmin: minor improvements to rrd4j-based pool-queue plots

There seems to be a bug in the rrd4j code. I have
reported this but am not confident anything will be done about it, since
version 2.2 came out in June of 2013 and no activity is evident
since then. In reality, the bug has already been discovered,
but the original reporter did not follow up by providing
a concrete case or data for verification. This is what I provided.

https://code.google.com/p/rrd4j/issues/detail?id=44

In essence, it seems that the stacked graphs are subject to
ghost artifacts which create a different colored border around
the area; if the time steps are narrow enough, this actually
can look as though it were displaying non-zero values for
a different data set. See the attached for comparison of
24, 48 and 96-hour graphs of the same data.

In the course of investigating this problem, I did discover
that the code was allowing unfeasible mappings of minimum
timesteps to fractions of a pixel. This has now been fixed
by automatically adjusting the width; the code also forces
at least two pixels per timestep (cfr. the 96-hour graph)
by expanding the width. This at least guarantees a certain
amount of legibility and mitigates the border-bug.

A number of other minor improvements and adjustments were
made:

-    The histogram has been slightly rewritten for efficiency;
-    An interface has been added to facilitate (1).
-    Use of the source name derived from the enum has been regularized.
-    "!" has been appended to this name, to enforce, according to rrd4j
    	convention, the conversion of NaNs to 0.0s.
-    The separate refresh interval has been eliminated. The sampling should
    	be congruent with the minimum time step.
-    Local variables have been added for re-used values.
-    The plot display names were being set to ALL CAPS for aesthetics/readability;
    	this has been removed to favor easy cut & paste into the filter
    	from other sources such as the logs or directory listings.
-    Default values for time span have been changed to something more reasonable
    	(48 hours rather than two weeks).

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Acked-by: Karsten
Require-book: no
Require-notes: yes

NOTE: The refresh property need not be removed in the backports (it will
just be ingored by the code).

Testing: Redeployed and observed plotting behavior (see attached) on testbed.

RELEASE NOTES:

Small adjustments to the pool queue plots in order to improve
appearance.  Some property defaults have been changed.  Note:
this patch does not entirely remedy what appears to be a bug
in rrd4j (see https://code.google.com/p/rrd4j/issues/detail?id=44).
billing: Add billing format for PoolHitInfoMessage

Otherwise the Indexer is unable to parse these billing entries and then
appear as empty records in the json and yaml output.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7355/
billing: Reduce startup time of dcache billing command

The patch also moves the class path utility functions to loadConfig as
they are not related to dcache domains/services.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7356/
billing: Improve error message on failur to parse date arguments

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: https://rb.dcache.org/r/7359/
billing: Add file size to request records

Request records submitted by doors lack the file size (they always
appeared as 0 in billing records). This patch adds the file size for
all protocols except for NFS.

Target: trunk
Request: 2.10
Request: 2.9
Request: 2.8
Request: 2.7
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7360/
billing: Fix date handling for json and yaml output

The raw date format in billing files lacks a year. The billing utility
has worked around that by prefixing the output with the full date. For
JSON and YAML output, this leads to a faulty timestamps like this:

    Jan 1, 2200: 01.01 00:00:00

This patch resolves the problem by converting it to an ISO8601 compliant
format (the preferred format for JSON and YAML).

The raw output is also adjusted such that the default format is prefixed
by the year rather than the full date.

JSON and YAML output formats have beeen adjusted to skip unrecognized
records. For YAML, the raw billing line is included as a comment.

Finally, the patch fixes an issue with outputing JSON from multiple days.

Target: trunk
Request: 2.10
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: https://rb.dcache.org/r/7357/
nfs4:  shutdown state handler on destroy

for graceful dcache shutdown

Target: master, 2.10
Acked-By: Paul Millar
Require-book: no
Require-notes: no
pool: ensure that mover error message always set

replace null objects with empty strings. update
all calls to iAbstractMover#setTransferStatus to
set non null message.

Acked-by: Paul Millar
Acked-by: Karsten Schwank
Target: master
Require-book: no
Require-notes: no