libs: Upgrade to JGlobus 2.0.6-rc3.d

Addresses a compatibility issue with some versions of curl and other
clients that send empty SSL messages.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5659/
gplazma2-grid: Fix user name regression in authzdb plugin

Addresses the issue that authzdb may add a second UserNamePrincipal and thus
cause the validation phase of a login to fail. This is a regression introduced
in 2.6.1.

The patch restores the old behaviour. The logic is not exactly beatiful, but
given that space manager relies on authorization by user name, it makes sense
that the user name is consistent with the UID. Hence if UID is selected by
something different that user name, then we have to replace the user name.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Patch: http://rb.dcache.org/r/5660/
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
logging: Set remote appender log level to off by default

Unless explicitly configured, we should not attempt to send any log events to
the remote appender (both for performance reasons and as a workaround for the
problem with the socket appender connecting to itself).

Target: trunk
Request: 2.6
Require-book: yes
Require-notes: yes
Patch: http://rb.dcache.org/r/5664/
Acked-by: Albert Rossi <arossi@fnal.gov>
Get rid of  java.sql.SQLException: Already have 1 record(s) with pnfsPath=.... for good

commit 635a05a78debe247c717b6fd27c18067af676ccb
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Fri Jun 21 15:18:27 2013 -0500

    spacemanager: Get rid of java.sql.SQLException: Already have 1 record(s) with pnfsPath=.... for good

    what happened

    Ticket: 7888
    Acked-by: Al Rossi
    Target: trunk
    Request: 2.6
    Request: 2.2
    Require-book: no
    Require-notes: no
gplazma: Fix DN extraction from proxy certificates

Addresses an issue in the x509 and xacml plugins in which the DN of
a proxy certificate rather the the DN of the user is extracted.

Extracting the DN of the user owning a proxy certificate involves
traversing the certificate chain, discarding any proxy certificates,
until the user's real certificate is found. The existing code in
dCache is not aware of all possible proxy certificate types, which
in some cases lead it to extract the DN of a proxy certififcate
rather than the user's DN.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5667/
cells: Remove "Sending CellException to" log message

Addresses an issue of frequent "Sending CellException to" messages in log files.
This is a regression introduced when we removed the last pieces of the old
logging system from dCache. These messages used to be surpressed, but now
happen to be logged.

The patch improves the text of the log message and changes the log level to
debug.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5665/
dcache-webadmin (alarms): provide option for absence of alarms database

module: dcache-webadmin, dcache (alarms)

The ticket reported an SQL stack trace error in the httpd log under the condition that they (a) are not running an alarmserver, and (b) have left the default settings for alarms alone.

The immediate fix was for them to set alarms.store.db.type=xml, but it makes sense for there to be an "off" option on this property as well.

This patch adds this option and makes it the default.  This complements the change of the remote log level default to off as well.

Testing done:

The following configurations were checked:

1.  type = off, alarmserver started
    (a) alarmserver reports: 21 Jun 2013 09:29:53 (alarms) [] alarms database type is OFF; server will not be started
    (b) httpd reports: nothing
    (c) page: The datastore for alarms and warnings has been disabled (see the configuration alarm properties).

2.  type = xml, no store.xml, alarmserver not started
    (b) httpd reports: nothing
    (c) page: The datastore for alarms and warnings has been disabled (see the configuration alarm properties).

3.  type = xml, alarmserver started
    everything is sane

4.  type = xml, with store.xml, alarmserver not started
    (b) httpd reports: nothing
    (c) page: query returns only what is currently in db.

5.  type = rdbms, no alarms db; alarmserver not started
    (b) httpd reports the underlying DataNucleus stack traces for "alarms database does not exist"
    (c) page: reports Internal error
    I would argue this is the correct behavior.  It is a misconfiguration.

6.  type = rdbms, no alarms db; alarmserver started
    (a) alarmserver reports Failed to create bean 'liquibase' : FATAL: database "alarms" does not exist from ac_create_$_2_3
        stops and tries to restart
    (b)/(c) same as above
    Again, this is correct behavior.

7.  type = rdbms, alarmserver not started
    (b) httpd reports: nothing
    (c) page: query returns only what is currently in db.

8.  type = rdbms, alarmserver started
    everything is sane

Target: master
Patch: http://rb.dcache.org/r/5668
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7887
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Gerd

RELEASE NOTES:
BOOK:

The options for alarms.store.db.type are not off,xml,rdbms, with off being the default.  If no alarms service is desired, the off option will prevent the web page from trying to establish a connection the the database.
(dcache-webadmin) null guard on billing service method

module: dcache-webadmin (billing)

An unexpected error in the service endpoint (billing cell) can result in the return value being null.  Rather than have a NPE reported, a guard as been added.

Target: master
Patch: http://rb.dcache.org/r/5670/
Require-notes: no
Require-book: no
Request: 2.6
Acked-by: Gerd
webdav: Fix problems with root paths

Addresses several issues in the webdav door:

- HTTP PUT without redirect is broken in earlier versions of 2.6. This
  regression was introduced in b460302934a6dab2cde1202a38883552d62ac65d.

  The problem is that the the resource path after upload is produced by
  concatenating the server's root path with a full dCache name space path,
  thus the root path appear twice in the generated path.

- The URI name space depends on who authenticates, which means that the
  same URI can have several meanings. This makes bookmarking and linking
  difficult.

- The CSS refers to a non-existing background image.

The solution to these problems are related. The patch removes the
webdavChrootToUserRoot property. An authenticated users root directory
is now always enforced, but only by checking that the request is within
the users root path. If a user has a non-null root and accesses the
root of the webdav door, a redirect to the users home directory is
generated.

The above change resulted in repeated authentication requests in the
browser, because the browser would try to access a non-existing
background image and a non-existing favicon.ico file. Access to these
would be denied, as they are outside the users root directory, and
thus the browser would ask for another user name and password. To
solve this problem, the patch fixes the CSS and adds a favicon.ico file.

An earlier version of this patch was submitted as http://rb.dcache.org/r/5201/

Target: trunk
Request: 2.6
Require-book: yes
Require-notes: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5672/
Merge branch 'master' of https://github.com/dCache/dcache
dcache: (remote logging) change default server port to be outside ephemeral port range

module: dcache (alarms)

This patch is an added precaution in connection with http://rb.dcache.org/r/5664.  It was discovered that the default port suggested by ch.qos.logback for the SocketAppender and server, 60001, is in fact not a good choice because it lies within the ephemeral port range for some OS's.  When there is no endpoint listening (I believe this only occurs when the server is supposed to be running local to the client domain), that port can be chosen as source for establishing a connection, such that the appender then ends up connecting to itself.  If events are actually sent over this connection, it quickly fills up the buffer and blocks.  This causes all threads which do logging to block, effectively freezing the system.

The above-mentioned patch sets remote logging by default to off, so that events are not sent unless it is explicitly activated.  The self-connected appender, however, is still potentially present (though at the 10 second reconnect timeout -- which is what the logback.xml is configured for -- it will take a while for it to find the 60001 port).

Setting the default port outside most ephemeral port ranges will guarantee that this connect-back does not happen.

Testing done:

Using a small test class with the remote logger reconnect delay set to 1 ms, and no server endpoint running, port 60001 was first chosen.  After 30 seconds,

[root@otfrid dcache]# netstat -anp | grep 60001
tcp        4      0 127.0.0.1:60001             127.0.0.1:60001             ESTABLISHED 6051/java

was seen.  The test waits 60 seconds, then begins to log continuously. The system freezes after 5446 events:

6001 25 Jun 2013 08:03:24 [] error message 1
...
6001 25 Jun 2013 08:03:24 [] error message 5446

Rerunning with port set to 9876 shows that after 10 minutes and more than 17M events, no connection is established and hence no freeze.

Target: master
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Gerd
Bug: http://rt.dcache.org/Ticket/Display.html?id=7856

RELEASE NOTES:

The default port for remote logging has been changed from 60001 to 9876; the latter lies outside most OS ephemeral port ranges.  This avoids a situation in which the connection host is localhost (the default), but there is no server listening on that port, thus permitting that port to be chosen as source for the appender connection and consequently establishing a connection to itself.  When this happens, the buffer quickly fills and blocks further logging, freezing all threads that do logging (effectively the entire domain).
cells: Fix login manager login limit interpretation

Addresses an issue in FTP and DCAP doors in which the login limit
was would allow n + 1 logins when n was given as a limit. Also
addresses an issue in which a limit of zero was considered to
be no limit at all. The correct value for disabling the limit is
-1.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5676/
info-provider: Escape percent sign in entity values

Percent signs are the only characters not allowed literally
in entity values, thus these must be escaped when exporting
the dCache configuration to an XML DTD.

The patch addresses the problem that info-provider is broken
by a recent patch adding logback formatting strings as
configuration values.

Target: trunk
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5673/
poolmanager, doors: Don't retry stage requests on the same pool

Addresses a regression introduced between 1.9.12 and 2.2. The regression
means that poolmanager would not remember the previously tried stage pool
and thus could select the same pool twice in a row when retrying failed
stage requests.

The patches addresses this problem. To be effective, both pool manager
and doors have to be updated, although the patch does not break any
version compatibility.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5678/
srm: Implement SRM_FILE_BUSY response

Addresses the issue, that the SRM is ignorant to other SRM requests
it may have on the same SURL. This leads to the dreaded "Already have 1
record" error in space management, as two concurrent puts on the same
file will be allowed by the SRM, but fail in space manager. More over,
there is currently no way for the user to abort the earlier requests if
they do not have the request number.

The SRM spec says that an upload operation should cause most other
operations on that SURL to fail with SRM_FILE_BUSY. srmRm should cause
existing get and put operations on the SURL to be aborted. This patch
implements those aspects of the SRM spec.

The patch does not implement this logic for srmCopy. This will be done
in a separate patch.

Since the patch addresses a real world issue we (and probably other
sites) suffer from, I request this to be merged into 2.6. Given the
change in observable behaviour, I am reluctant to merge this into 2.2
at this point.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5677/
dcache-webadmin: avoid rrd4j timestep error statement

module: dcache-webadmin (pool queue plots)

Under certain restart conditions, the rrd4j utility will report that the call to write is doing so with a timestamp which is less than 1 second since the last update.  While this error is not fatal, it is annoying to see in the log.

The extra check simply avoids writing the data under these conditions and waits till the next iteration.

TESTING DONE:

Without the patch, rapid multiple restarts of the webadmin service/httpd domain will probably produce a trace similar to this:

26 Jun 2013 09:32:47 (System) [] problem running collectPoolSeclectionUnit
java.lang.IllegalArgumentException: Bad sample time: 1372257000. Last update time was 1372257000, at least one second step is required
        at org.rrd4j.core.RrdDb.store(RrdDb.java:553) ~[rrd4j-2.1.1.jar:na]
        at org.rrd4j.core.Sample.update(Sample.java:197) ~[rrd4j-2.1.1.jar:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.storeToRRD(RrdPoolInfoAgent.java:331) ~[classes/:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.processPools(RrdPoolInfoAgent.java:291) ~[classes/:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.run(RrdPoolInfoAgent.java:138) ~[classes/:na]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0]

With the patch, the log remains clean during multiple successive attempts to restart.

Target: master
Patch: http://rb.dcache.org/r/5680
Require-notes: yes
Require-book: no
Request: 2.6
Acked-by: Gerd

RELEASE NOTES:  The cause of the stack trace occasionally seen in the webadmin log file (on restart of the service) has been addressed.
pinmanager, other: Make pinmanager configuration comply with new policy

Addresses the issue that pin manager properties follow inconsistent
naming conventions.

This patch applies the naming conventions agreed to at the 2013 developers
workshop, with a few personal additions. The patch also affects most other
services, as some changed properties are used by most of them.

Guiding principals:

- All properties used by a service are prefixed by the service name (pinmanager
  in this case).

- All properties are lower case

- Properties are structured hierarchically

- A dash is used as a word separator

- Time units are made explicit by adding a property suffixes with .unit

- Properties that are used by many services have an equivalent property with
  the dcache. prefix.

- Old properties are kept during a trandition period, but marked deprecated.

- Eventually the scoping operator will be removed, but during a transition period
  it is still used. This is to avoid chaging all properties in one big bang.

New configuration families introduced:

service.<service> defines properties for communicating with other cells. Available
in both the shared dcache. hierarchy and in a per service hierarchy (currently only
pinmanager). Not for properties belonging to that service. This is only for properties
other services use to talk to <service>.

cell.threads. for properties related to threading of cell (in particular threads for
message delivery).

cell.queue. for properties related to message queueing.

authz. for authorization related properties.

Properties introduced:

dcache.service.pnfsmanager
dcache.service.poolmanager
dcache.service.billing
dcache.service.gplazma
dcache.service.spacemanager
dcache.service.pinmanager
dcache.service.replica
dcache.service.loginbroker
dcache.authz.staging
dcache.authz.staging.pep
pinmanager.cell.name
pinmanager.cell.threads.min
pinmanager.cell.threads.max
pinmanager.cell.threads.max-idle-time
pinmanager.cell.threads.max-idle-time.unit
pinmanager.cell.queue.max
pinmanager.service.poolmanager
pinmanager.service.poolmanager.timeout
pinmanager.service.poolmanager.timeout.unit
pinmanager.service.pnfsmanager
pinmanager.service.pnfsmanager.timeout
pinmanager.service.pnfsmanager.timeout.unit
pinmanager.service.pool.timeout
pinmanager.service.pool.timeout.unit
pinmanager.db.host
pinmanager.db.name
pinmanager.db.user
pinmanager.db.password
pinmanager.db.password.file
pinmanager.db.driver
pinmanager.db.url
pinmanager.db.schema.changelog
pinmanager.db.connections.max-per-partition
pinmanager.db.connections.min-per-partition
pinmanager.expiration-period
pinmanager.expiration-period.unit
pinmanager.max-pin-duration
pinmanager.max-pin-duration.unit
pinmanager.authz.staging
pinmanager.authz.staging.pep

Properties deprecated:

pnfsmanager
poolmanager
billing
gplazma
spacemanager
loginBroker
stageConfigurationFilePath
stagePolicyEnforcementPoint
pinManagerDbHost
pinManagerDbName
pinManagerDbUser
pinManagerDbPassword
pinManagerPasswordFile
pinManagerMaxActiveJdbcConnections
pinManagerMaxPinDuration

Target: trunk
Require-notes: yes
Require-book: yes
Merge branch 'master' of https://github.com/dCache/dcache
xrootd: Make xrootd configuration comply with new policy

Changes spill over to other services too, but the main focus of
the patch is to rename all xrootd properties.

Introduces:

dcache.service.loginbroker.update-period
dcache.service.loginbroker.update-period.unit
dcache.service.loginbroker.update-threshold
pool.mover.xrootd.timeout.idle
xrootd.cell.name
xrootd.net.port
xrootd.net.backlog
xrootd.net.listen
xrootd.threads
xrootd.max-total-memory-size
xrootd.max-channel-memory-size
xrootd.service.poolmanager
xrootd.service.poolmanager.timeout
xrootd.service.poolmanager.timeout.unit
xrootd.service.pnfsmanager
xrootd.service.gplazma
xrootd.service.gplazma.timeout
xrootd.service.gplazma.timeout.unit
xrootd.service.billing
xrootd.service.loginbroker
xrootd.service.loginbroker.update-period
xrootd.service.loginbroker.update-period.unit
xrootd.service.loginbroker.update-threshold
xrootd.service.loginbroker.family
xrootd.service.loginbroker.version
xrootd.service.pool.timeout
xrootd.service.pool.timeout.unit
xrootd.root
xrootd.authz.read-paths
xrootd.authz.write-paths
xrootd.authz.user
xrootd.plugins
xrootd.mover.queue
xrootd.mover.timeout
xrootd.mover.timeout.unit
admin.net.listen
dcache.net.listen
dcap.net.listen
gsidcap.net.listen
kerberosdcap.net.listen
authdcap.net.listen
dcap.service.loginbroker
dcap.service.loginbroker.update-period
dcap.service.loginbroker.update-period.unit
dcap.service.loginbroker.update-threshold
authdcap.service.loginbroker
authdcap.service.loginbroker.update-period
authdcap.service.loginbroker.update-period.unit
authdcap.service.loginbroker.update-threshold
kerberosdcap.service.loginbroker
kerberosdcap.service.loginbroker.update-period
kerberosdcap.service.loginbroker.update-period.unit
kerberosdcap.service.loginbroker.update-threshold
gsidcap.service.loginbroker
gsidcap.service.loginbroker.update-period
gsidcap.service.loginbroker.update-period.unit
gsidcap.service.loginbroker.update-threshold
ftp.net.listen
gridftp.net.listen
kerberosftp.net.listen
ftp.service.loginbroker
ftp.service.loginbroker.update-period
ftp.service.loginbroker.update-period.unit
ftp.service.loginbroker.update-threshold
gridftp.service.loginbroker
gridftp.service.loginbroker.update-period
gridftp.service.loginbroker.update-period.unit
gridftp.service.loginbroker.update-threshold
kerberosftp.service.loginbroker
kerberosftp.service.loginbroker.update-period
kerberosftp.service.loginbroker.update-period.unit
kerberosftp.service.loginbroker.update-threshold
nfsv41.service.loginbroker
nfsv41.service.loginbroker.update-period
nfsv41.service.loginbroker.update-period.unit
nfsv41.service.loginbroker.update-threshold
srm.net.listen
srm.service.loginbroker
srm.service.loginbroker.update-period
srm.service.loginbroker.update-period.unit
srm.service.loginbroker.update-threshold
webdav.net.listen
webdav.service.loginbroker
webdav.service.loginbroker.update-period
webdav.service.loginbroker.update-period.unit
webdav.service.loginbroker.update-threshold

Deprecated:

loginBrokerUpdateTime
loginBrokerUpdateThreshold
xrootdPort
xrootd/port
xrootdBacklog
xrootdThreads
xrootdMaxTotalMemorySize
xrootdMaxChannelMemorySize
xrootdPoolManagerTimeout
xrootdPoolTimeout
xrootdRootPath
xrootdAllowedWritePaths
xrootdAllowedReadPaths
xrootdPlugins
xrootdUser
xrootdIoQueue
xrootdMoverTimeout
listen
srm.loginbroker
srm.loginbroker.period
srm.loginbroker.threshold

Removed (previously deprecated):

xrootdAuthNPlugin
xrootdAuthzPlugin

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5642/
poolmanager, pool: Drop classic partition and most space cost related code

This patch drops the classic partition. Wass has been the default partition
type for new partitions for a while already. During that time we have not
heard about any issues with wass. Therefore this patch drops the classic
partition type.

Technically, ClassicPartition is kept as a base class for WassPartition, but
the dead code is dropped. A lot of related code for computing the space
cost is dropped as well as it is no longer used.

A number of user visible changes should be documented in the release notes:

- Sites still using the classic partition type must manually update
  poolmanager.conf to create wass partitions instead.

- When switching from classic to wass, it is important to reset cpucostfactor
  and spacecostfactor, as the interpretation of these parameters has changed.
  We suggest to use the default value of 1.0 for both parameters.

- The alternate PoolSelectionUnitV2wgap has been dropped. It can no longer
  be used as a value for the selectionUnit configuration property. We believe
  that this functionality is no longer required when using wass. Should a
  need for such functionality arise in the future, it would better be
  implemented as a new partition type rather than a selection unit.

- The pool manager 'xcm ls' command no longer accepts any arguments or
  options.

- The pool manager 'cm ls' command no longer accepts the -size and -d
  options.

- The pool manager 'getpoolsbylink' command no longer accepts the -size
  option.

- The pool migration module no longer recognizes the source.spaceCost and
  target.spaceCost variables.

- The pool migration module no longer accepts 'best' as a value for
  the -select option.

- The pool manager 'rebalance pgroup' command no longer accepts 'sc' as a
  value for the -metric option. The new value 'free' has been added instead.

- The pool manager 'costCalculation' configuration parameter is obsolete.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5682/