nfs: added enable/disable switch to embeded portmat

In some cases we do not want to start embedded portmap
even if OS does not provide one.

introduced property

nfs.enable.portmap=true/false

Default it true for backward compatibility

Acked-by: Gerd Behrmann
Target: trunk
Require-book: no
Require-notes: yes
libdcap: remove redundant code

The dcap code is since middle of January 2013 has
its own repository:

https://github.com/dCache/dcap

Acked-by: Gerd Behrmann
Target: trunk
Require-book: no
Require-notes: no
libs: Upgrade to JGlobus 2.0.6-rc3.d

Addresses a compatibility issue with some versions of curl and other
clients that send empty SSL messages.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5659/
gplazma2-grid: Fix user name regression in authzdb plugin

Addresses the issue that authzdb may add a second UserNamePrincipal and thus
cause the validation phase of a login to fail. This is a regression introduced
in 2.6.1.

The patch restores the old behaviour. The logic is not exactly beatiful, but
given that space manager relies on authorization by user name, it makes sense
that the user name is consistent with the UID. Hence if UID is selected by
something different that user name, then we have to replace the user name.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Patch: http://rb.dcache.org/r/5660/
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
logging: Set remote appender log level to off by default

Unless explicitly configured, we should not attempt to send any log events to
the remote appender (both for performance reasons and as a workaround for the
problem with the socket appender connecting to itself).

Target: trunk
Request: 2.6
Require-book: yes
Require-notes: yes
Patch: http://rb.dcache.org/r/5664/
Acked-by: Albert Rossi <arossi@fnal.gov>
Get rid of  java.sql.SQLException: Already have 1 record(s) with pnfsPath=.... for good

commit 635a05a78debe247c717b6fd27c18067af676ccb
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Fri Jun 21 15:18:27 2013 -0500

    spacemanager: Get rid of java.sql.SQLException: Already have 1 record(s) with pnfsPath=.... for good

    what happened

    Ticket: 7888
    Acked-by: Al Rossi
    Target: trunk
    Request: 2.6
    Request: 2.2
    Require-book: no
    Require-notes: no
gplazma: Fix DN extraction from proxy certificates

Addresses an issue in the x509 and xacml plugins in which the DN of
a proxy certificate rather the the DN of the user is extracted.

Extracting the DN of the user owning a proxy certificate involves
traversing the certificate chain, discarding any proxy certificates,
until the user's real certificate is found. The existing code in
dCache is not aware of all possible proxy certificate types, which
in some cases lead it to extract the DN of a proxy certififcate
rather than the user's DN.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5667/
cells: Remove "Sending CellException to" log message

Addresses an issue of frequent "Sending CellException to" messages in log files.
This is a regression introduced when we removed the last pieces of the old
logging system from dCache. These messages used to be surpressed, but now
happen to be logged.

The patch improves the text of the log message and changes the log level to
debug.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5665/
dcache-webadmin (alarms): provide option for absence of alarms database

module: dcache-webadmin, dcache (alarms)

The ticket reported an SQL stack trace error in the httpd log under the condition that they (a) are not running an alarmserver, and (b) have left the default settings for alarms alone.

The immediate fix was for them to set alarms.store.db.type=xml, but it makes sense for there to be an "off" option on this property as well.

This patch adds this option and makes it the default.  This complements the change of the remote log level default to off as well.

Testing done:

The following configurations were checked:

1.  type = off, alarmserver started
    (a) alarmserver reports: 21 Jun 2013 09:29:53 (alarms) [] alarms database type is OFF; server will not be started
    (b) httpd reports: nothing
    (c) page: The datastore for alarms and warnings has been disabled (see the configuration alarm properties).

2.  type = xml, no store.xml, alarmserver not started
    (b) httpd reports: nothing
    (c) page: The datastore for alarms and warnings has been disabled (see the configuration alarm properties).

3.  type = xml, alarmserver started
    everything is sane

4.  type = xml, with store.xml, alarmserver not started
    (b) httpd reports: nothing
    (c) page: query returns only what is currently in db.

5.  type = rdbms, no alarms db; alarmserver not started
    (b) httpd reports the underlying DataNucleus stack traces for "alarms database does not exist"
    (c) page: reports Internal error
    I would argue this is the correct behavior.  It is a misconfiguration.

6.  type = rdbms, no alarms db; alarmserver started
    (a) alarmserver reports Failed to create bean 'liquibase' : FATAL: database "alarms" does not exist from ac_create_$_2_3
        stops and tries to restart
    (b)/(c) same as above
    Again, this is correct behavior.

7.  type = rdbms, alarmserver not started
    (b) httpd reports: nothing
    (c) page: query returns only what is currently in db.

8.  type = rdbms, alarmserver started
    everything is sane

Target: master
Patch: http://rb.dcache.org/r/5668
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7887
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Gerd

RELEASE NOTES:
BOOK:

The options for alarms.store.db.type are not off,xml,rdbms, with off being the default.  If no alarms service is desired, the off option will prevent the web page from trying to establish a connection the the database.
(dcache-webadmin) null guard on billing service method

module: dcache-webadmin (billing)

An unexpected error in the service endpoint (billing cell) can result in the return value being null.  Rather than have a NPE reported, a guard as been added.

Target: master
Patch: http://rb.dcache.org/r/5670/
Require-notes: no
Require-book: no
Request: 2.6
Acked-by: Gerd
webdav: Fix problems with root paths

Addresses several issues in the webdav door:

- HTTP PUT without redirect is broken in earlier versions of 2.6. This
  regression was introduced in b460302934a6dab2cde1202a38883552d62ac65d.

  The problem is that the the resource path after upload is produced by
  concatenating the server's root path with a full dCache name space path,
  thus the root path appear twice in the generated path.

- The URI name space depends on who authenticates, which means that the
  same URI can have several meanings. This makes bookmarking and linking
  difficult.

- The CSS refers to a non-existing background image.

The solution to these problems are related. The patch removes the
webdavChrootToUserRoot property. An authenticated users root directory
is now always enforced, but only by checking that the request is within
the users root path. If a user has a non-null root and accesses the
root of the webdav door, a redirect to the users home directory is
generated.

The above change resulted in repeated authentication requests in the
browser, because the browser would try to access a non-existing
background image and a non-existing favicon.ico file. Access to these
would be denied, as they are outside the users root directory, and
thus the browser would ask for another user name and password. To
solve this problem, the patch fixes the CSS and adds a favicon.ico file.

An earlier version of this patch was submitted as http://rb.dcache.org/r/5201/

Target: trunk
Request: 2.6
Require-book: yes
Require-notes: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5672/
Merge branch 'master' of https://github.com/dCache/dcache
dcache: (remote logging) change default server port to be outside ephemeral port range

module: dcache (alarms)

This patch is an added precaution in connection with http://rb.dcache.org/r/5664.  It was discovered that the default port suggested by ch.qos.logback for the SocketAppender and server, 60001, is in fact not a good choice because it lies within the ephemeral port range for some OS's.  When there is no endpoint listening (I believe this only occurs when the server is supposed to be running local to the client domain), that port can be chosen as source for establishing a connection, such that the appender then ends up connecting to itself.  If events are actually sent over this connection, it quickly fills up the buffer and blocks.  This causes all threads which do logging to block, effectively freezing the system.

The above-mentioned patch sets remote logging by default to off, so that events are not sent unless it is explicitly activated.  The self-connected appender, however, is still potentially present (though at the 10 second reconnect timeout -- which is what the logback.xml is configured for -- it will take a while for it to find the 60001 port).

Setting the default port outside most ephemeral port ranges will guarantee that this connect-back does not happen.

Testing done:

Using a small test class with the remote logger reconnect delay set to 1 ms, and no server endpoint running, port 60001 was first chosen.  After 30 seconds,

[root@otfrid dcache]# netstat -anp | grep 60001
tcp        4      0 127.0.0.1:60001             127.0.0.1:60001             ESTABLISHED 6051/java

was seen.  The test waits 60 seconds, then begins to log continuously. The system freezes after 5446 events:

6001 25 Jun 2013 08:03:24 [] error message 1
...
6001 25 Jun 2013 08:03:24 [] error message 5446

Rerunning with port set to 9876 shows that after 10 minutes and more than 17M events, no connection is established and hence no freeze.

Target: master
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Gerd
Bug: http://rt.dcache.org/Ticket/Display.html?id=7856

RELEASE NOTES:

The default port for remote logging has been changed from 60001 to 9876; the latter lies outside most OS ephemeral port ranges.  This avoids a situation in which the connection host is localhost (the default), but there is no server listening on that port, thus permitting that port to be chosen as source for the appender connection and consequently establishing a connection to itself.  When this happens, the buffer quickly fills and blocks further logging, freezing all threads that do logging (effectively the entire domain).
cells: Fix login manager login limit interpretation

Addresses an issue in FTP and DCAP doors in which the login limit
was would allow n + 1 logins when n was given as a limit. Also
addresses an issue in which a limit of zero was considered to
be no limit at all. The correct value for disabling the limit is
-1.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5676/
info-provider: Escape percent sign in entity values

Percent signs are the only characters not allowed literally
in entity values, thus these must be escaped when exporting
the dCache configuration to an XML DTD.

The patch addresses the problem that info-provider is broken
by a recent patch adding logback formatting strings as
configuration values.

Target: trunk
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5673/
poolmanager, doors: Don't retry stage requests on the same pool

Addresses a regression introduced between 1.9.12 and 2.2. The regression
means that poolmanager would not remember the previously tried stage pool
and thus could select the same pool twice in a row when retrying failed
stage requests.

The patches addresses this problem. To be effective, both pool manager
and doors have to be updated, although the patch does not break any
version compatibility.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5678/
srm: Implement SRM_FILE_BUSY response

Addresses the issue, that the SRM is ignorant to other SRM requests
it may have on the same SURL. This leads to the dreaded "Already have 1
record" error in space management, as two concurrent puts on the same
file will be allowed by the SRM, but fail in space manager. More over,
there is currently no way for the user to abort the earlier requests if
they do not have the request number.

The SRM spec says that an upload operation should cause most other
operations on that SURL to fail with SRM_FILE_BUSY. srmRm should cause
existing get and put operations on the SURL to be aborted. This patch
implements those aspects of the SRM spec.

The patch does not implement this logic for srmCopy. This will be done
in a separate patch.

Since the patch addresses a real world issue we (and probably other
sites) suffer from, I request this to be merged into 2.6. Given the
change in observable behaviour, I am reluctant to merge this into 2.2
at this point.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5677/
dcache-webadmin: avoid rrd4j timestep error statement

module: dcache-webadmin (pool queue plots)

Under certain restart conditions, the rrd4j utility will report that the call to write is doing so with a timestamp which is less than 1 second since the last update.  While this error is not fatal, it is annoying to see in the log.

The extra check simply avoids writing the data under these conditions and waits till the next iteration.

TESTING DONE:

Without the patch, rapid multiple restarts of the webadmin service/httpd domain will probably produce a trace similar to this:

26 Jun 2013 09:32:47 (System) [] problem running collectPoolSeclectionUnit
java.lang.IllegalArgumentException: Bad sample time: 1372257000. Last update time was 1372257000, at least one second step is required
        at org.rrd4j.core.RrdDb.store(RrdDb.java:553) ~[rrd4j-2.1.1.jar:na]
        at org.rrd4j.core.Sample.update(Sample.java:197) ~[rrd4j-2.1.1.jar:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.storeToRRD(RrdPoolInfoAgent.java:331) ~[classes/:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.processPools(RrdPoolInfoAgent.java:291) ~[classes/:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.run(RrdPoolInfoAgent.java:138) ~[classes/:na]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0]

With the patch, the log remains clean during multiple successive attempts to restart.

Target: master
Patch: http://rb.dcache.org/r/5680
Require-notes: yes
Require-book: no
Request: 2.6
Acked-by: Gerd

RELEASE NOTES:  The cause of the stack trace occasionally seen in the webadmin log file (on restart of the service) has been addressed.
pinmanager, other: Make pinmanager configuration comply with new policy

Addresses the issue that pin manager properties follow inconsistent
naming conventions.

This patch applies the naming conventions agreed to at the 2013 developers
workshop, with a few personal additions. The patch also affects most other
services, as some changed properties are used by most of them.

Guiding principals:

- All properties used by a service are prefixed by the service name (pinmanager
  in this case).

- All properties are lower case

- Properties are structured hierarchically

- A dash is used as a word separator

- Time units are made explicit by adding a property suffixes with .unit

- Properties that are used by many services have an equivalent property with
  the dcache. prefix.

- Old properties are kept during a trandition period, but marked deprecated.

- Eventually the scoping operator will be removed, but during a transition period
  it is still used. This is to avoid chaging all properties in one big bang.

New configuration families introduced:

service.<service> defines properties for communicating with other cells. Available
in both the shared dcache. hierarchy and in a per service hierarchy (currently only
pinmanager). Not for properties belonging to that service. This is only for properties
other services use to talk to <service>.

cell.threads. for properties related to threading of cell (in particular threads for
message delivery).

cell.queue. for properties related to message queueing.

authz. for authorization related properties.

Properties introduced:

dcache.service.pnfsmanager
dcache.service.poolmanager
dcache.service.billing
dcache.service.gplazma
dcache.service.spacemanager
dcache.service.pinmanager
dcache.service.replica
dcache.service.loginbroker
dcache.authz.staging
dcache.authz.staging.pep
pinmanager.cell.name
pinmanager.cell.threads.min
pinmanager.cell.threads.max
pinmanager.cell.threads.max-idle-time
pinmanager.cell.threads.max-idle-time.unit
pinmanager.cell.queue.max
pinmanager.service.poolmanager
pinmanager.service.poolmanager.timeout
pinmanager.service.poolmanager.timeout.unit
pinmanager.service.pnfsmanager
pinmanager.service.pnfsmanager.timeout
pinmanager.service.pnfsmanager.timeout.unit
pinmanager.service.pool.timeout
pinmanager.service.pool.timeout.unit
pinmanager.db.host
pinmanager.db.name
pinmanager.db.user
pinmanager.db.password
pinmanager.db.password.file
pinmanager.db.driver
pinmanager.db.url
pinmanager.db.schema.changelog
pinmanager.db.connections.max-per-partition
pinmanager.db.connections.min-per-partition
pinmanager.expiration-period
pinmanager.expiration-period.unit
pinmanager.max-pin-duration
pinmanager.max-pin-duration.unit
pinmanager.authz.staging
pinmanager.authz.staging.pep

Properties deprecated:

pnfsmanager
poolmanager
billing
gplazma
spacemanager
loginBroker
stageConfigurationFilePath
stagePolicyEnforcementPoint
pinManagerDbHost
pinManagerDbName
pinManagerDbUser
pinManagerDbPassword
pinManagerPasswordFile
pinManagerMaxActiveJdbcConnections
pinManagerMaxPinDuration

Target: trunk
Require-notes: yes
Require-book: yes
Merge branch 'master' of https://github.com/dCache/dcache