pinmanager, other: Make pinmanager configuration comply with new policy

Addresses the issue that pin manager properties follow inconsistent
naming conventions.

This patch applies the naming conventions agreed to at the 2013 developers
workshop, with a few personal additions. The patch also affects most other
services, as some changed properties are used by most of them.

Guiding principals:

- All properties used by a service are prefixed by the service name (pinmanager
  in this case).

- All properties are lower case

- Properties are structured hierarchically

- A dash is used as a word separator

- Time units are made explicit by adding a property suffixes with .unit

- Properties that are used by many services have an equivalent property with
  the dcache. prefix.

- Old properties are kept during a trandition period, but marked deprecated.

- Eventually the scoping operator will be removed, but during a transition period
  it is still used. This is to avoid chaging all properties in one big bang.

New configuration families introduced:

service.<service> defines properties for communicating with other cells. Available
in both the shared dcache. hierarchy and in a per service hierarchy (currently only
pinmanager). Not for properties belonging to that service. This is only for properties
other services use to talk to <service>.

cell.threads. for properties related to threading of cell (in particular threads for
message delivery).

cell.queue. for properties related to message queueing.

authz. for authorization related properties.

Properties introduced:

dcache.service.pnfsmanager
dcache.service.poolmanager
dcache.service.billing
dcache.service.gplazma
dcache.service.spacemanager
dcache.service.pinmanager
dcache.service.replica
dcache.service.loginbroker
dcache.authz.staging
dcache.authz.staging.pep
pinmanager.cell.name
pinmanager.cell.threads.min
pinmanager.cell.threads.max
pinmanager.cell.threads.max-idle-time
pinmanager.cell.threads.max-idle-time.unit
pinmanager.cell.queue.max
pinmanager.service.poolmanager
pinmanager.service.poolmanager.timeout
pinmanager.service.poolmanager.timeout.unit
pinmanager.service.pnfsmanager
pinmanager.service.pnfsmanager.timeout
pinmanager.service.pnfsmanager.timeout.unit
pinmanager.service.pool.timeout
pinmanager.service.pool.timeout.unit
pinmanager.db.host
pinmanager.db.name
pinmanager.db.user
pinmanager.db.password
pinmanager.db.password.file
pinmanager.db.driver
pinmanager.db.url
pinmanager.db.schema.changelog
pinmanager.db.connections.max-per-partition
pinmanager.db.connections.min-per-partition
pinmanager.expiration-period
pinmanager.expiration-period.unit
pinmanager.max-pin-duration
pinmanager.max-pin-duration.unit
pinmanager.authz.staging
pinmanager.authz.staging.pep

Properties deprecated:

pnfsmanager
poolmanager
billing
gplazma
spacemanager
loginBroker
stageConfigurationFilePath
stagePolicyEnforcementPoint
pinManagerDbHost
pinManagerDbName
pinManagerDbUser
pinManagerDbPassword
pinManagerPasswordFile
pinManagerMaxActiveJdbcConnections
pinManagerMaxPinDuration

Target: trunk
Require-notes: yes
Require-book: yes
Merge branch 'master' of https://github.com/dCache/dcache
xrootd: Make xrootd configuration comply with new policy

Changes spill over to other services too, but the main focus of
the patch is to rename all xrootd properties.

Introduces:

dcache.service.loginbroker.update-period
dcache.service.loginbroker.update-period.unit
dcache.service.loginbroker.update-threshold
pool.mover.xrootd.timeout.idle
xrootd.cell.name
xrootd.net.port
xrootd.net.backlog
xrootd.net.listen
xrootd.threads
xrootd.max-total-memory-size
xrootd.max-channel-memory-size
xrootd.service.poolmanager
xrootd.service.poolmanager.timeout
xrootd.service.poolmanager.timeout.unit
xrootd.service.pnfsmanager
xrootd.service.gplazma
xrootd.service.gplazma.timeout
xrootd.service.gplazma.timeout.unit
xrootd.service.billing
xrootd.service.loginbroker
xrootd.service.loginbroker.update-period
xrootd.service.loginbroker.update-period.unit
xrootd.service.loginbroker.update-threshold
xrootd.service.loginbroker.family
xrootd.service.loginbroker.version
xrootd.service.pool.timeout
xrootd.service.pool.timeout.unit
xrootd.root
xrootd.authz.read-paths
xrootd.authz.write-paths
xrootd.authz.user
xrootd.plugins
xrootd.mover.queue
xrootd.mover.timeout
xrootd.mover.timeout.unit
admin.net.listen
dcache.net.listen
dcap.net.listen
gsidcap.net.listen
kerberosdcap.net.listen
authdcap.net.listen
dcap.service.loginbroker
dcap.service.loginbroker.update-period
dcap.service.loginbroker.update-period.unit
dcap.service.loginbroker.update-threshold
authdcap.service.loginbroker
authdcap.service.loginbroker.update-period
authdcap.service.loginbroker.update-period.unit
authdcap.service.loginbroker.update-threshold
kerberosdcap.service.loginbroker
kerberosdcap.service.loginbroker.update-period
kerberosdcap.service.loginbroker.update-period.unit
kerberosdcap.service.loginbroker.update-threshold
gsidcap.service.loginbroker
gsidcap.service.loginbroker.update-period
gsidcap.service.loginbroker.update-period.unit
gsidcap.service.loginbroker.update-threshold
ftp.net.listen
gridftp.net.listen
kerberosftp.net.listen
ftp.service.loginbroker
ftp.service.loginbroker.update-period
ftp.service.loginbroker.update-period.unit
ftp.service.loginbroker.update-threshold
gridftp.service.loginbroker
gridftp.service.loginbroker.update-period
gridftp.service.loginbroker.update-period.unit
gridftp.service.loginbroker.update-threshold
kerberosftp.service.loginbroker
kerberosftp.service.loginbroker.update-period
kerberosftp.service.loginbroker.update-period.unit
kerberosftp.service.loginbroker.update-threshold
nfsv41.service.loginbroker
nfsv41.service.loginbroker.update-period
nfsv41.service.loginbroker.update-period.unit
nfsv41.service.loginbroker.update-threshold
srm.net.listen
srm.service.loginbroker
srm.service.loginbroker.update-period
srm.service.loginbroker.update-period.unit
srm.service.loginbroker.update-threshold
webdav.net.listen
webdav.service.loginbroker
webdav.service.loginbroker.update-period
webdav.service.loginbroker.update-period.unit
webdav.service.loginbroker.update-threshold

Deprecated:

loginBrokerUpdateTime
loginBrokerUpdateThreshold
xrootdPort
xrootd/port
xrootdBacklog
xrootdThreads
xrootdMaxTotalMemorySize
xrootdMaxChannelMemorySize
xrootdPoolManagerTimeout
xrootdPoolTimeout
xrootdRootPath
xrootdAllowedWritePaths
xrootdAllowedReadPaths
xrootdPlugins
xrootdUser
xrootdIoQueue
xrootdMoverTimeout
listen
srm.loginbroker
srm.loginbroker.period
srm.loginbroker.threshold

Removed (previously deprecated):

xrootdAuthNPlugin
xrootdAuthzPlugin

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5642/
poolmanager, pool: Drop classic partition and most space cost related code

This patch drops the classic partition. Wass has been the default partition
type for new partitions for a while already. During that time we have not
heard about any issues with wass. Therefore this patch drops the classic
partition type.

Technically, ClassicPartition is kept as a base class for WassPartition, but
the dead code is dropped. A lot of related code for computing the space
cost is dropped as well as it is no longer used.

A number of user visible changes should be documented in the release notes:

- Sites still using the classic partition type must manually update
  poolmanager.conf to create wass partitions instead.

- When switching from classic to wass, it is important to reset cpucostfactor
  and spacecostfactor, as the interpretation of these parameters has changed.
  We suggest to use the default value of 1.0 for both parameters.

- The alternate PoolSelectionUnitV2wgap has been dropped. It can no longer
  be used as a value for the selectionUnit configuration property. We believe
  that this functionality is no longer required when using wass. Should a
  need for such functionality arise in the future, it would better be
  implemented as a new partition type rather than a selection unit.

- The pool manager 'xcm ls' command no longer accepts any arguments or
  options.

- The pool manager 'cm ls' command no longer accepts the -size and -d
  options.

- The pool manager 'getpoolsbylink' command no longer accepts the -size
  option.

- The pool migration module no longer recognizes the source.spaceCost and
  target.spaceCost variables.

- The pool migration module no longer accepts 'best' as a value for
  the -select option.

- The pool manager 'rebalance pgroup' command no longer accepts 'sc' as a
  value for the -metric option. The new value 'free' has been added instead.

- The pool manager 'costCalculation' configuration parameter is obsolete.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5682/
spacemanager : improve selecPool logic a little

commit 0bca233fba745c862b6c7253bfa94bb1945c241c
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Fri Jun 28 11:46:00 2013 -0500

    spacemanager : improve selecPool logic a little

    what happened

    Ticket: #number
    Acked-by: Gerd Behrmann
    Target: trunk
    Require-book: no
    Require-notes: no

commit 3a4537c5fa4f0484d01aaba3ad881c94d44673f9
Author: Dmitry Litvintsev <litvinse@fnal.gov>
Date:   Mon Jun 24 16:01:16 2013 -0500

    module: summary line
    streamline selectPool logic a little

    what happened

    Ticket: #number
    Acked-by:
    Target: trunk
    Require-book: no
    Require-notes: no
pinmanager: Fix typo in pin manager property

Target: trunk
Require-notes: no
Require-book: no
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7900
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5685/
Fix various database-related niggles

This patch fixes a number of relatively minor database-related irritations:

  1. When dCache starts, the SRM's AuthRecord persistency BoneCP db-pool emits a

         Max Connections < 1. Setting to 20

     warning.  This is fixed by specifying the maximum number of connections.

  2. Remove the unnecessary 'provider' from JDO persistence-unit stanza (we're no
     longer using TopLink, so this work-around is not needed any more).

  3. Fix various unit-tests to configure BoneCP's max connections, so they no
     longer emit the "Max Connections < 1" warning.

  4. Remove a dead-code line from database initialisation (used by space-manager
     and replica-manager).  The dead-code is likely due to a typo ("Min" vs "Max")
     but, to avoid possible side-effects, this patch simply removes the dead-code.

  5. Fix AuthRecordPersistenceUnit configuration so that DataNucleus' (JPA) enhancer
     no longer attempts to modify the byte-code of all dCache classes, but only
     those necessary.  This makes the build *much* less noisy and slightly
     faster.

When the DataNucleus' (JDO) enhancer modifies classes for the pin-manager, only a single
class needs enhancing (org.dcache.pinmanager.model.Pin), however the enhancer reports
that some 30 classes were considered.  I don't know which 30 classes are examined, nor
why the enhancer is examining classes other than Pin.

Patch: http://rb.dcache.org/r/5687/
Acked-by: Albert Rossi
Target: trunk
Requires-notes: no
Requires-book: no
webdav: Comply with new property policy

New

missing-files.timeout.unit
webdav.cell.name
webdav.service.pnfsmanager
webdav.service.pnfsmanager.timeout
webdav.service.pnfsmanager.timeout.unit
webdav.service.poolmanager
webdav.service.poolmanager.timeout
webdav.service.poolmanager.timeout.unit
webdav.service.spacemanager.enabled
webdav.service.pool.timeout
webdav.service.pool.timeout.unit
webdav.service.gplazma
webdav.service.gplazma.timeout
webdav.service.gplazma.timeout.unit
webdav.service.billing
webdav.service.loginbroker.version
webdav.service.loginbroker.family
webdav.service.missing-files
webdav.service.missing-files.timeout
webdav.service.missing-files.timeout.unit
webdav.service.missing-files.enabled
webdav.mover.queue
webdav.mover.kill-timeout
webdav.mover.kill-timeout.unit
webdav.mover.timeout
webdav.mover.timeout.unit
webdav.mover.transfer-confirmation-timeout
webdav.mover.transfer-confirmation-timeout.unit
webdav.net.port
webdav.net.internal
webdav.root
webdav.authz.allowed-paths
webdav.authz.readonly
webdav.authz.anonymous-operations
webdav.authz.anonymous-listing
webdav.authn.protocol
webdav.authn.keystore
webdav.authn.keystore.password
webdav.authn.truststore
webdav.authn.truststore.password
webdav.authn.accept-client-cert
webdav.authn.require-client-cert
webdav.authn.basic
webdav.authn.hostcert.cert
webdav.authn.hostcert.key
webdav.authn.hostcert.key.unit
webdav.authn.capath
webdav.authn.capath.refresh
webdav.authn.capath.refresh.unit

Deprecated

webdavPnfsTimeout
webdavPoolManagerTimeout
webdavPoolTimeout
webdavGplazmaTimeout
webdavIoQueue
webdavKillTimeout
webdavMoverTimeout
webdavTransferConfirmationTimeout
webdavPort
webdav/port
webdavInternalAddress
webdavRootPath
webdavAllowedPaths
webdavReadOnly
webdavAnonymousAccess
webdavAnonymousListing
webdavProtocol
webdavKeyStore
webdavKeyStorePassword
webdavTrustStore
webdavTrustStorePassword
webdavWantClientAuth
webdavNeedClientAuth
webdavBasicAuthentication

Removed

hostCertificateRefreshPeriod
trustAnchorRefreshPeriod
verifyHostCertificateChain
webdav.images.logo
webdav.images.directory
webdav.images.file
webdav.style.css
webdavLogoPath
webdavDirIconPath
webdavFileIconPath
webdavCssPath
logoPath
dirIconPath
fileIconPath
cssPath
webdav.templates.list
httpPortNumber
httpRootPath
httpAllowedPaths
webdavContextPath

Other changes

srmSpaceManagerEnabled only accepts the values true and false

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5683/
Fix nsswitch plugin unit-tests

The nsswitch gPlazma plugin uses JNA to access some native functions
common to various BSD-like OSes (including GNU/Linux).  It implements
the identity-, map- and session- gPlazma plugin functionality.  The
identity- and map- functionality is based on username <--> uid,
group-name <--> gid and a user's group-membership.  The session
functionality provides hard-coded behaviour (which might be better
factored out as a separate plugin).

There are several problems with the current nsswitch unit-tests:

  1. The unit-tests depend on JNA.  Not all JVMs support JNA, so can
     cause problems, especially since JNA relies on a native library,
     which is not available for all OSes.

  2. The unit-tests depend on OS/native functions.  Not all OSes
     support these functions.  While this prevents such OSes from
     using the nsswitch plugin for deployed dCache instances, it
     should not prevent developers from using such OSes.

  3. The unit-tests depend on system configuration.  While most
     Unix-like systems have a user "root" with uid 0 and primary gid
     0; other users and groups are not standardised --- even gid 0 can
     have different group-name ("root" or "wheel")

  4. The existing tests cover only part of the identity functionality.
     The tests do not cover group--gid identity-reverse-mapping, all
     map and session functionality.

This patch addresses these issues by fully mocking the JNA-backed
classes.  That way, the unit-tests are completely isolated from JNA,
the underlying OS and its configuration.  This allows a complete
coverage of the nsswitch plugin's functionality in a portable fashion.

It turns out that mocking the JNA-based classes that nsswitch plugin
uses is particularly tricky as these Java classes inherit from system
classes that make use of static initialisers and make JNI calls from
the constructor.  Mockito-mocked classes solve points 2. to 4., but
will still require a valid JNA, so point 1. is still a problem.

To solve point 1., this patch adds the PowerMock framework.  This is
an extension to mockito that adds support for the byte-code
manipulation needed to avoid static initializer and suppress
superclass constructors.  The nsswitch unit-tests are still written
using junit and mockito; however, in addition, PowerMock is used as
required.  Note that all existing unit-tests continue to function as
normal and the extra PowerMock functionality is only invoked when the
@RunWith(PowerMockRunner.class) annotation is specified.

The existing nsswitch tests are rewritten as BDD tests and additional
tests are added.  Fluent classes and utility methods are used make the
tests as readable as possible, at the risk of somewhat obfuscating
what is actually going on.

Target: master
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5688/
Acked-by: Gerd Behrmann
Teach git to ignore generated content

Maven adds considerable predictability of where generated files are
stored.  These are of little interest to any SCM since they are
managed by the build process and the results should not be stored in a
SCM.

This patch adds a .gitignore file that excludes the 'target'
directories in the current modules (or sub-projects) from git's
consideration.  Following maven convention, this should be sufficient
to exclude all auto-generated material.

Target: trunk
Requires-notes: no
Requires-book: no
Patch: http://rb.dcache.org/r/5686/
Acked-by: Gerd Behrmann
Acked-by: Albert Rossi
ftp: Comply with new property policy

Besides updating all ftp related properties to comply with the
new policy, this patch collapses the various ftp services into
a single service. A property is now used to select the authentication
protocol. This is how it is done for webdav and xrootd already. The
new configuration policy encourages this approach as most ftp
properties are independent of the authentication protocol and were
shared by all serices. Collapsing them into a single service makes
the service more DRY.

New properties:

ftp.cell.name
ftp.authn.protocol
ftp.authn.kerberos.service-principle-name
ftp.authn.kerberos.key-distribution-center-list
ftp.authn.hostcert.key
ftp.authn.hostcert.cert
ftp.authn.capath
ftp.authn.kpwd
ftp.authz.stage
ftp.authz.stage.pep
ftp.authz.readonly
ftp.net.port
ftp.net.port-range
ftp.net.internal
ftp.performance-marker-period
ftp.performance-marker-period.unit
ftp.limits.retries
ftp.limits.streams-per-client
ftp.limits.clients
ftp.mover.queue
ftp.transaction-log
ftp.overwrite
ftp.security.ciphers
ftp.service.poolmanager
ftp.service.poolmanager.timeout
ftp.service.poolmanager.timeout.unit
ftp.service.spacemanager.enabled
ftp.service.pool.timeout
ftp.service.pool.timeout.unit
ftp.service.pnfsmanager
ftp.service.pnfsmanager.timeout
ftp.service.pnfsmanager.timeout.unit
ftp.service.loginbroker.family
ftp.service.loginbroker.version
ftp.service.gplazma
ftp.service.billing

Deprecated

ftpPort
kerberosFtpPort
gsiFtpPortNumber
clientDataPortRange
gsiftpAdapterInternalInterface
performanceMarkerPeriod
gsiftpMaxRetries
gsiftpMaxStreamsPerClient
gsiftpMaxLogin
gsiftpDeleteOnConnectionClosed
gsiftpIoQueue
FtpTLogDir
gridftp.security.ciphers
gsiftpPoolManagerTimeout
gsiftpPoolManager
gsiftpPoolTimeout
gsiftpPnfsTimeout

Forbidden

ftp/kerberos.service-principle-name
ftp.read-only
ftp/port

Removed

ftpBase
spaceReservation
spaceReservationStrict
gsiftpAllowPassivePool
kerberosScvPrincipal
gsiftpDefaultStreamsPerClient

Removed services

kerberosftp
gridftp

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5694/
(logging) minor adjustments to error-level logging (@author arossi)

I have gone through all the .java files carrying the annotation @author arossi, looking for logging irregularities.  I have tried to eliminate things like logger.error logging a stack trace when not clearly an unexpected exception (bug).  There are also a few other minor fixes.

Target: master
Patch: http://rb.dcache.org/r/5690/
Require-notes: no
Require-book: no
Acked-by: Paul
dcache: (billing) Fix incorrect MoverData field name

Regression introduced by last large multi-patch billing update.

The MoverData field 'p2p' was renamed to 'isPtp', following suggestion that Java convention be followed.  However, the liquibase and package.jdo files continued to assume the database table attribute was 'p2p'.  Hence when the data object gets dehydrated by DataNucleus, the insert statement fails.

This patch reverts the Java field name to 'p2p'.

Testing done:

Without the patch, writes to the billing database will produce the following error:

02 jul 2013 05:43:03 (billing) [] Insert of object "org.dcache.services.billing.db.data.MoverData@6e19f13" using statement "INSERT INTO "billinginfo"
("transfersize","fullsize","cellname","datestamp","protocol","action","errormessage","initiator","transaction","storageclass","errorcode","client","isp2p","pnfsid","isnew","connectiontime")
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : user lacks privilege or object not found: isp2p
02 jul 2013 05:43:03 (billing) [] could not not commit; some billing
data may be lost

With the patch, successful srmcp produced the correct entry in billinginfo.

Note:  this has been tested on a regular instance of dcache, not using system-test.

Target: master
Patch: http://rb.dcache.org/r/5692/
Require-book: no
Require-notes: yes
Request: 2.6
Acked-by: Gerd

RELEASE NOTES:  Fixes bug preventing inserts into the billinginfo table.