srm: Make login broker of SRM configurable and clean up properties

The patch allows the login brokers used by the SRM to be configurable.
Since the SRM uses two login brokers, one for lookup and one for
registration, I have renamed all related properties such that those
related to door lookup begin with srm.lookup. and those for login
broker registration begin with srm.loginbroker.

When merging into stable branches, the old properties will not be marked
deprecated.

Deprecated by this patch:

srmLoginBrokerUpdatePeriod
srmNumberOfDoorsInRandomSelection

Introduced by this patch:

srm.lookup.loginbroker
srm.lookup.timeout
srm.lookup.population-size
srm.loginbroker
srm.loginbroker.period
srm.loginbroker.threshold
srm.loginbroker.family
srm.loginbroker.version

Target: trunk
Request: 2.6
Request: 2.2
Require-book: yes
Require-notes: yes
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5604/
Merge branch 'master' of https://github.com/dCache/dcache
srm: Make login broker of SRM configurable and clean up properties

Fixed small bug introduced in 7654683528daebdb246b82db1b4d73562b4cc1c9.

Target: trunk
Request: 2.6
Request: 2.2
Require-book: no
Require-notes: no
Patch: http://rb.dcache.org/r/5604
Added billing property

Specifies the cell address services submit billing info to. Not all
services respect this property (at least statistics and poolmanager
don't).

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5605/
poolmanager: Made billing cell address configurable for pool manager

Addresses the issue that the billing cell address for pool manager
was injected through admin commands (rc set warning path) rather
than through the configuration system.

Obsoletes the admin command 'rc set warning path'.

Target: trunk
Require-notes: yes
Require-book: yes
Patch: http://rb.dcache.org/r/5606/
Acked-by: Paul Millar <paul.millar@desy.de> (in person)
Acked-by: Tigran Mkrtchya <tigran.mkrtchyan@desy.de>
pool: Improve logging of common errors

Improve log levels and log messages of common errors found in
production log files.

Target: trunk
Require-book: no
Require-notes: yes
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7685
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvinsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5255/
dcache: alarms fix four small bugs

module dcache alarms

Four small bugs:

1.  alarm send:  if -s does not provide a path, the subsequent string operation throws array out of bounds.
2.  alarm definition toXML is not emitting the matchException node when the value is true rather than false (the default)
3.  the server logback file defines the MAIL appender reference after it uses it, which is not allowed.
4.  If an alarm has an id which permits duplicates, and the admin closes the alarm, but thereafter another of its type arrives, that alarm currently will not be visible unless show closed is selected.  The fix is to force the closed flag to false on duplicate arrival.

Target: master
Patch:  http://rb.dcache.org/r/5602
Require-notes: yes
Require-book: no
Request: 2.6
Acked-by: Tigran

RELEASE NOTES:

Fixes four small bugs.

1.  dcache alarm send:  if -s does not provide a path (e.g., "src://otfrid" rather than "src://otfrid/"), the subsequent string o$
2.  dcache alarm add * modify fail to record matchException = true because of a logic error in the XML emitter.
3.  the server logback file defined the MAIL appender reference after it uses it, which is not allowed.
4.  If an alarm has an id which permits duplicates, and the admin closes the alarm, but thereafter another of its type arrives, that alarm currently will not be visible unless show closed is selected.  The fix is to force the closed flag to false on duplicate arrival.
pool: Remove caching behaviour of file meta backend

Revert 8290f2f878a818a1398bea7c415a5b1274ec47eb

Tigran reported that the pool performs badly with the
simple caching behaviour added in the above patch.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5608/
Merge branch 'master' of https://github.com/dCache/dcache
broadcast: Fix expiration of dynamic subscribers

Addresses the issue that HSM cleaner is broken as it newer
receives pool-up notifications.

BroadcastRegistrationTask supplies a relative timeout, but the
broadcast registration cell expects an absolute timeout.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5601/
gplazma: Simplify mapping plugin interface

Addresses an issue with the interface of gPlazma mapping plugins.
Third party mapping plugins will have to be updated.

The old interface distinguished between identified principals and
authorized principals. The idea was that only principals that had
somehow been recognized by a mapping plugin would be considered
authorized and returned to dCache. This approach howeve has several
problems:

- Complexity

- Redundancy; a principal by definition is an assertion that the
  principal belongs to the user (although the level of trust can
  be given a principal strongly depends on its type, eg a DN is
  stronger that a user name, and a user name is stronger than an
  unchecked login name). More over, a principal in it self doesn't
  give access to any additional resources: A second authorization
  step is always done by dCache, authorizing a particular principal
  to do something.

- Limiting, as it was impossible to just "pass through" a principal
  already generated and confirmed by the door. Some doors can do
  their own authentication and there is no reason why one should
  absolutely have to configure a mapping plugin just for the i
  principal to be passed through.

- Inability of authentication plugins to generate principals that
  are passed through to dCache without further mapping. For instance,
  the XACML plugin takes DN and FQAN extracted from the cert chain
  and does a lookup through XACML to generate a user name. A
  subsequent mapping plugin maps the user name to UID/GID, but the
  DN and FQAN is lost as it is not mapped. Yet both DN and FQAN are
  used by other components in dCache for authorization steps.

This patch addresses the problem by changing the mapping plugin
interface. It now takes a single set of principals which it can
add principals to (and possible remove from). Tests are updated
to reflect the changed semantics.

Merge for 2.6 is requested because the patch is a prerequisite to
resolve the XACML problem mentioned above.

Target: master
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5545/
chimera: update tags mtime on re-write

keep nfs client cache happy

Acked-by: Dmitry Litvintsev
Patch: http://rb.dcache.org/r/5617/
Target: trunk, 2.6, 2.2
Require-book: no
Require-notes: no
webdav: Fix interpretation of client cert properties

Addresses a problem that causes the interpretation of the
webdavWantClientAuth and webdavNeedClientAuth properties to
be reversed.

JGlobus 2 seems to have a different interpretation of these
flags. The relevant JGlobus code is:

	 if (this.requireClientAuth.booleanValue() == Boolean.TRUE) {
             this.sslEngine.setNeedClientAuth(this.requireClientAuth.booleanValue());
	 } else
             this.sslEngine.setWantClientAuth(!this.acceptNoClientCerts.booleanValue());

Target: trunk
Request: 2.6
Request: 2.2-sha2
Require-notes: yes
Require-book: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5615/
Add support for banning problematic ciphers for Jetty SSL connectors

An earlier patch added support for banning ciphers through the JGlobus
library.  The same problem exists for the connectors supplied with
Jetty, which are used in the WebDAV/HTTP door and the webadmin.

This patch adds support for banning problematic ciphers for
these services.

Target: master
Request: 2.6
Signed-off-by: Paul Millar <paul.millar@desy.de>
Acked-by: Gerd Behrmann <behrmann@nordu.net>
Patch: http://rb.dcache.org/r/5614/
Require-notes: yes
Require-book: no
scripts: fix invalid path in dcache-pool-meta-preupgrade

Acked-by: Gerd Behrmann
Target: trunk, 2.6
Require-book: no
Require-notes: no
chimera:  fix corruption of directory mode on tag update

in case up create/update of a directory tag, the directory mode was
corrupted as  setInodeAttributes was used.

This is a simple fix before the complete set/get attribute rewrite.

Acked-by: Gerd Behrmann
Target: trunk, 2.6
Require-book: no
Require-notes: yes
pool: Improve resilience against broken SI files

Addresses an issue in which corrupt SI files could cause pool
startup to fail. Eg. NPE or OOM during deserialization would
cause the pool to fail and it would not even log the file
triggering the problem. Such exceptions would typically be
caused by corrupt length fields or similar invalid data in
the SI file.

Also improved the exception chaining and logging in this part
of the pool.

The problem was reported on the user-forum list.

Target: trunk
Request: 2.6
Request: 2.2
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5609/
ftp: Remove duplicate session ID in NDC

Addresses the issue that the session ID is stored twice in the diagnostic
context of log messages generated by the FTP door.

The NDC is already initialized with the session ID in Tranfer#initSession.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5610/
billing: restructure to use service-oriented architecture

Combines four patches.

5415: Modify billing DAO to support p2p attribute and new request API (2)

module: core billing db/dao

Part 2.

This patch builds on http://rb.dcache.org/r/5413 to address the following:

A.  Previous versions of the billing plot generation relied on the placement of the generating cell (billing) on the same file partition as the web server.  This has made the billing plots unusable at DESY.

B.  In the current billing data tables there is no way to track migration/pool-to-pool operations as distinct from door-initiated uploads and downloads.

In support of these,

- An interface for timestamped data objects (ITimestamped) has been moved out of the billing package in core and placed in common; a second interface, IHistogramData, replaces IPlotData.  Plotting and histogram data are thus conceptually separated.

- The DAO objects have undergone small redesigning and extensions for p2p operations

- General cleanup and adjustment of main DAO implementation class

- Liquibase changelog to modify the tables

- Standalone migration script for updating data with p2p attribute values.

-----------------------------------------------------------------------------

5416: Add billing request API, handlers, and histogram implementation (3)

module: billing database and cell

Part 3.

This patch refactors the billing functionality so that it now is service-oriented.  In order to do so, we have added a general API layer for remote requests to the billing cell, providing one concrete implementation, that for histogram data.  The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction (such that plot generation can take place elsewhere, on the calling end).

New interfaces and data objects have been provided which allow for the wire transmission of an implementation-neutral data wrapper for histogram data; a service interface supporting this transfer is also implemented, with the client using the synchronous cell endpoint sendAndWait call to talk to the service, which in turn takes the request and interprets it via introspection in order invoke a method on the IBillingInfoAccess interface and return the query result to the caller.

The histogram request semantics are now the layer at which the specific JDO queries are articulated.

As stated, this implementation is actually based on a more general API for BillingRequests and their handlers, so that additional handlers could be added in the future.  The handlers are mapped by the BillingDatabase to the return type of the request object they are responsible for.

-----------------------------------------------------------------------------

5417: Refactor Jaida plot modules based on new request API (4)

module: billing plots

Part 4.

The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction.  This patch refactors plotting against the new APIs.  It is still implemented using JAIDA.  While it still resides in the dcache core module, it is actually used by the webadmin service.

We have also modified what is and what is not exposed in terms of plot configuration.  Aside from the choices for style and scale, property configuration is internal.  plotPropertiesFile path is no longer used.

The plot generation interfaces, along with the underlying JAIDA implementation, have be extended to produce the new p2p plots.

The plots generated currently use connected data points and a log scale; however, DESY has plots which use an bar/outline style. We would like to allow the installation or the user to display the plots using a few simple options; hence, we have added options and default properties for them:

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  (These options are actually made available via user-facing choices on the new billing plots page; see the final patch in this series).

-----------------------------------------------------------------------------

5578: Changes to webadmin to accommodate new billing SOA; no plot rendering options (5)
module: webadmin billing

Part 5.  Changes to webadmin in connection with the reworked billing infrastructure.

Version of 5418 in which the plot rendering options are not exposed to the user.

Conversion to implementation-specific histograms and plots (in the current case, using the org.freehep packages) now takes place on the receiving end of the call.  The JAIDA-based classes are included in the core module, but are now utilized inside of the webadmin container.

This is accomplished by adding a service abstraction and implementation to the controller package of webadmin; this service is used by the billing page.

Refresh can be achieved (a) by forcing it with the refresh button, (b) leaving the window open Ajax auto-refresh is added to the form, run once every refresh interval) and/or (c), a daemon thread in the service which runs at the refresh interval.

Property configuration is internal.  plotPropertiesFile path is no longer used.

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  Once the default is selected, this is the form in which the service serves up the plots to everyone.

-----------------------------------------------------------------------------

Testing:

For the redesign, a fully redeployed RPM was tested with the billing cell running both inside and outside the httpd domain.  Tests for the following conditions:

    - reload with no plots
    - reload with unexpired plots
    - reload with expired plots
    - force refresh

were run in Firefox, Safari and Chrome.

Also tested were the default settings for the style and scaling, and switching between those options.

For p2p, multiple migrations done from pool to pool through admin interface and logs/database/plots checked for consistency.

-----------------------------------------------------------------------------

Patch: http://rb.dcache.org/r/5415
Patch: http://rb.dcache.org/r/5416
Patch: http://rb.dcache.org/r/5417
Patch: http://rb.dcache.org/r/5578
Acked-by: Gerd
Target: master
Require-book: yes
Require-notes: yes
Request: 2.6

RELEASE NOTES:
BOOK:
Draw attention to the possibility of adding a clause in the accounting logs (format for mover messages): '[p2p=true/false]'.  The database schema for the billinginfo table has been modified to include this boolean 'p2p' indicating a pool-to-pool transfer.  An sql script for migrating old data (an optional procedure to make the billing table consistent which will require several hours of downtime) is included (/usr/share/dcache/migration/migrate_to_billing_with_p2p.sql).

The billing cell now is independent of the domain where the plots are rendered (the webadmin service); there is no longer a requirement to run billing on the same file partition as the httpd domain.  note that "billingToDb" needs to be visible only to the billing service, and "generatePlots" only to the httpd service.  Note that plotPropertiesFile is obsolete and no longer used. There several new properties for controlling plot types.

webadmin.billing-plot.refresh-threshold=5

webadmin.billing-plot.height=800

webadmin.billing-plot.width=1200

(one-of?CONNECTED|OUTLINE|FILLED)webadmin.billing-plot.default-style=CONNECTED

(one-of?lin|log)webadmin.billing-plot.default-scale=log
nfs: allow protocol family name be configurable

The protocol family name used by LoginBroker and published
as supported transfer protocol by SRM.

Target: trunk, 2.2, 2.6
Ticket: #7233, #7874
Require-notes: yes
Require-book: yes

Acked-by: Paul Millar
Acked-By: Gerd Behrmann
Patch: http://rb.dcache.org/r/5020