billing: restructure to use service-oriented architecture

Combines four patches.

5415: Modify billing DAO to support p2p attribute and new request API (2)

module: core billing db/dao

Part 2.

This patch builds on http://rb.dcache.org/r/5413 to address the following:

A.  Previous versions of the billing plot generation relied on the placement of the generating cell (billing) on the same file partition as the web server.  This has made the billing plots unusable at DESY.

B.  In the current billing data tables there is no way to track migration/pool-to-pool operations as distinct from door-initiated uploads and downloads.

In support of these,

- An interface for timestamped data objects (ITimestamped) has been moved out of the billing package in core and placed in common; a second interface, IHistogramData, replaces IPlotData.  Plotting and histogram data are thus conceptually separated.

- The DAO objects have undergone small redesigning and extensions for p2p operations

- General cleanup and adjustment of main DAO implementation class

- Liquibase changelog to modify the tables

- Standalone migration script for updating data with p2p attribute values.

-----------------------------------------------------------------------------

5416: Add billing request API, handlers, and histogram implementation (3)

module: billing database and cell

Part 3.

This patch refactors the billing functionality so that it now is service-oriented.  In order to do so, we have added a general API layer for remote requests to the billing cell, providing one concrete implementation, that for histogram data.  The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction (such that plot generation can take place elsewhere, on the calling end).

New interfaces and data objects have been provided which allow for the wire transmission of an implementation-neutral data wrapper for histogram data; a service interface supporting this transfer is also implemented, with the client using the synchronous cell endpoint sendAndWait call to talk to the service, which in turn takes the request and interprets it via introspection in order invoke a method on the IBillingInfoAccess interface and return the query result to the caller.

The histogram request semantics are now the layer at which the specific JDO queries are articulated.

As stated, this implementation is actually based on a more general API for BillingRequests and their handlers, so that additional handlers could be added in the future.  The handlers are mapped by the BillingDatabase to the return type of the request object they are responsible for.

-----------------------------------------------------------------------------

5417: Refactor Jaida plot modules based on new request API (4)

module: billing plots

Part 4.

The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction.  This patch refactors plotting against the new APIs.  It is still implemented using JAIDA.  While it still resides in the dcache core module, it is actually used by the webadmin service.

We have also modified what is and what is not exposed in terms of plot configuration.  Aside from the choices for style and scale, property configuration is internal.  plotPropertiesFile path is no longer used.

The plot generation interfaces, along with the underlying JAIDA implementation, have be extended to produce the new p2p plots.

The plots generated currently use connected data points and a log scale; however, DESY has plots which use an bar/outline style. We would like to allow the installation or the user to display the plots using a few simple options; hence, we have added options and default properties for them:

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  (These options are actually made available via user-facing choices on the new billing plots page; see the final patch in this series).

-----------------------------------------------------------------------------

5578: Changes to webadmin to accommodate new billing SOA; no plot rendering options (5)
module: webadmin billing

Part 5.  Changes to webadmin in connection with the reworked billing infrastructure.

Version of 5418 in which the plot rendering options are not exposed to the user.

Conversion to implementation-specific histograms and plots (in the current case, using the org.freehep packages) now takes place on the receiving end of the call.  The JAIDA-based classes are included in the core module, but are now utilized inside of the webadmin container.

This is accomplished by adding a service abstraction and implementation to the controller package of webadmin; this service is used by the billing page.

Refresh can be achieved (a) by forcing it with the refresh button, (b) leaving the window open Ajax auto-refresh is added to the form, run once every refresh interval) and/or (c), a daemon thread in the service which runs at the refresh interval.

Property configuration is internal.  plotPropertiesFile path is no longer used.

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  Once the default is selected, this is the form in which the service serves up the plots to everyone.

-----------------------------------------------------------------------------

Testing:

For the redesign, a fully redeployed RPM was tested with the billing cell running both inside and outside the httpd domain.  Tests for the following conditions:

    - reload with no plots
    - reload with unexpired plots
    - reload with expired plots
    - force refresh

were run in Firefox, Safari and Chrome.

Also tested were the default settings for the style and scaling, and switching between those options.

For p2p, multiple migrations done from pool to pool through admin interface and logs/database/plots checked for consistency.

-----------------------------------------------------------------------------

Patch: http://rb.dcache.org/r/5415
Patch: http://rb.dcache.org/r/5416
Patch: http://rb.dcache.org/r/5417
Patch: http://rb.dcache.org/r/5578
Acked-by: Gerd
Target: master
Require-book: yes
Require-notes: yes
Request: 2.6

RELEASE NOTES:
BOOK:
Draw attention to the possibility of adding a clause in the accounting logs (format for mover messages): '[p2p=true/false]'.  The database schema for the billinginfo table has been modified to include this boolean 'p2p' indicating a pool-to-pool transfer.  An sql script for migrating old data (an optional procedure to make the billing table consistent which will require several hours of downtime) is included (/usr/share/dcache/migration/migrate_to_billing_with_p2p.sql).

The billing cell now is independent of the domain where the plots are rendered (the webadmin service); there is no longer a requirement to run billing on the same file partition as the httpd domain.  note that "billingToDb" needs to be visible only to the billing service, and "generatePlots" only to the httpd service.  Note that plotPropertiesFile is obsolete and no longer used. There several new properties for controlling plot types.

webadmin.billing-plot.refresh-threshold=5

webadmin.billing-plot.height=800

webadmin.billing-plot.width=1200

(one-of?CONNECTED|OUTLINE|FILLED)webadmin.billing-plot.default-style=CONNECTED

(one-of?lin|log)webadmin.billing-plot.default-scale=log
Merge remote branch 'main/2.6' into feature/2.6/billing_soa

Conflicts:
	skel/share/defaults/dcache.properties
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
merged: http://rb.dcache.org/r/5641
merged: http://rb.dcache.org/r/5646
Parameterize log levels and log formats

The log levels and log formats are now customizable through
dcache.conf.

New configuration properties:

dcache.log.level.file
dcache.log.level.pinboard
dcache.log.level.remote
dcache.log.level.events
dcache.log.format.file
dcache.log.format.pinboard

Target: master
Require-notes: yes
Require-book: yes
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5530/
(cherry picked from commit a96423a403e3c4ad12f99d02191c3f80157a4549)
logging: Set remote appender log level to off by default

Unless explicitly configured, we should not attempt to send any log events to
the remote appender (both for performance reasons and as a workaround for the
problem with the socket appender connecting to itself).

Target: trunk
Request: 2.6
Require-book: yes
Require-notes: yes
Patch: http://rb.dcache.org/r/5664/
Acked-by: Albert Rossi <arossi@fnal.gov>
Fix quickJava for some shell versions

The current implementation and usage of quickJava relies on a shell-
specific behaviour; specifically, that specifying a environment
variable for a command that is actually a shell function becomes
an environment variable for that shell function.

Although this works for some versions of bash, it's now clear that
there are other versions of bash (and perhaps other shells) for
which this does not hold.

This patch fixes this problem by explicitly exporting the CLASSPATH
variable as an environment variable.

Patch: http://rb.dcache.org/r/5603
Acked-by: Tigran Mkrtchyan
Target: master
Requires-book: no
Requires-notes: no
(cherry picked from commit 083a7eb7263186d4cc72d2e8123459ab286d81e7)
Respect session read-only in SRM

Ticket: 7842
Acked-by: Al Rossi
Target: 2.6
Require-book: no
Require-notes: yes
gplazma: Fix DN extraction from proxy certificates

Addresses an issue in the x509 and xacml plugins in which the DN of
a proxy certificate rather the the DN of the user is extracted.

Extracting the DN of the user owning a proxy certificate involves
traversing the certificate chain, discarding any proxy certificates,
until the user's real certificate is found. The existing code in
dCache is not aware of all possible proxy certificate types, which
in some cases lead it to extract the DN of a proxy certififcate
rather than the user's DN.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5667/
(cherry picked from commit b848b326b54eec0b5196febbff1201a65d1403d7)
cells: Remove "Sending CellException to" log message

Addresses an issue of frequent "Sending CellException to" messages in log files.
This is a regression introduced when we removed the last pieces of the old
logging system from dCache. These messages used to be surpressed, but now
happen to be logged.

The patch improves the text of the log message and changes the log level to
debug.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5665/
(cherry picked from commit b419ba459fd57eeac1bd0af4554340436a364360)
(2.6) dcache-webadmin (alarms): provide option for absence of alarms database

module: dcache-webadmin, dcache (alarms)

The ticket reported an SQL stack trace error in the httpd log under the condition that they (a) are not running an alarmserver, and (b) have left the default settings for alarms alone.

The immediate fix was for them to set alarms.store.db.type=xml, but it makes sense for there to be an "off" option on this property as well.

This patch adds this option and makes it the default.  This complements the change of the remote log level default to off as well.

Testing done:

The following configurations were checked:

1.  type = off, alarmserver started
    (a) alarmserver reports: 21 Jun 2013 09:29:53 (alarms) [] alarms database type is OFF; server will not be started
    (b) httpd reports: nothing
    (c) page: The datastore for alarms and warnings has been disabled (see the configuration alarm properties).

2.  type = xml, no store.xml, alarmserver not started
    (b) httpd reports: nothing
    (c) page: The datastore for alarms and warnings has been disabled (see the configuration alarm properties).

3.  type = xml, alarmserver started
    everything is sane

4.  type = xml, with store.xml, alarmserver not started
    (b) httpd reports: nothing
    (c) page: query returns only what is currently in db.

5.  type = rdbms, no alarms db; alarmserver not started
    (b) httpd reports the underlying DataNucleus stack traces for "alarms database does not exist"
    (c) page: reports Internal error
    I would argue this is the correct behavior.  It is a misconfiguration.

6.  type = rdbms, no alarms db; alarmserver started
    (a) alarmserver reports Failed to create bean 'liquibase' : FATAL: database "alarms" does not exist from ac_create_$_2_3
        stops and tries to restart
    (b)/(c) same as above
    Again, this is correct behavior.

7.  type = rdbms, alarmserver not started
    (b) httpd reports: nothing
    (c) page: query returns only what is currently in db.

8.  type = rdbms, alarmserver started
    everything is sane

Target: master
Patch: http://rb.dcache.org/r/5668
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7887
Request: 2.6
Require-notes: yes
Require-book: yes
Acked-by: Gerd

RELEASE NOTES:
BOOK:

The options for alarms.store.db.type are not off,xml,rdbms, with off being the default.  If no alarms service is desired, the off option will prevent the web page from trying to establish a connection the the database.
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
Merged master patch http://rb.dcache.org/r/5670
webdav: Fix problems with root paths

Addresses several issues in the webdav door:

- HTTP PUT without redirect is broken in earlier versions of 2.6. This
  regression was introduced in b460302934a6dab2cde1202a38883552d62ac65d.

  The problem is that the the resource path after upload is produced by
  concatenating the server's root path with a full dCache name space path,
  thus the root path appear twice in the generated path.

- The URI name space depends on who authenticates, which means that the
  same URI can have several meanings. This makes bookmarking and linking
  difficult.

- The CSS refers to a non-existing background image.

The solution to these problems are related. The patch removes the
webdavChrootToUserRoot property. An authenticated users root directory
is now always enforced, but only by checking that the request is within
the users root path. If a user has a non-null root and accesses the
root of the webdav door, a redirect to the users home directory is
generated.

The above change resulted in repeated authentication requests in the
browser, because the browser would try to access a non-existing
background image and a non-existing favicon.ico file. Access to these
would be denied, as they are outside the users root directory, and
thus the browser would ask for another user name and password. To
solve this problem, the patch fixes the CSS and adds a favicon.ico file.

An earlier version of this patch was submitted as http://rb.dcache.org/r/5201/

Target: trunk
Request: 2.6
Require-book: yes
Require-notes: yes
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5672/
(cherry picked from commit e48a670d502815203704af59a2dedfa7307c13ba)

Conflicts:
	modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java
dcache: (remote logging) change default server port to be outside ephemeral port range

module: dcache (alarms)

This patch is an added precaution in connection with http://rb.dcache.org/r/5664.  It was discovered that the default port suggested by ch.qos.logback for the SocketAppender and server, 60001, is in fact not a good choice because it lies within the ephemeral port range for some OS's.  When there is no endpoint listening (I believe this only occurs when the server is supposed to be running local to the client domain), that port can be chosen as source for establishing a connection, such that the appender then ends up connecting to itself.  If events are actually sent over this connection, it quickly fills up the buffer and blocks.  This causes all threads which do logging to block, effectively freezing the system.

The above-mentioned patch sets remote logging by default to off, so that events are not sent unless it is explicitly activated.  The self-connected appender, however, is still potentially present (though at the 10 second reconnect timeout -- which is what the logback.xml is configured for -- it will take a while for it to find the 60001 port).

Setting the default port outside most ephemeral port ranges will guarantee that this connect-back does not happen.

Testing done:

Using a small test class with the remote logger reconnect delay set to 1 ms, and no server endpoint running, port 60001 was first chosen.  After 30 seconds,

[root@otfrid dcache]# netstat -anp | grep 60001
tcp        4      0 127.0.0.1:60001             127.0.0.1:60001             ESTABLISHED 6051/java

was seen.  The test waits 60 seconds, then begins to log continuously. The system freezes after 5446 events:

6001 25 Jun 2013 08:03:24 [] error message 1
...
6001 25 Jun 2013 08:03:24 [] error message 5446

Rerunning with port set to 9876 shows that after 10 minutes and more than 17M events, no connection is established and hence no freeze.

Target: 2.6
Require-notes: yes
Require-book: no
Acked-by: Gerd
Bug: http://rt.dcache.org/Ticket/Display.html?id=7856
Merge-req:

RELEASE NOTES:

The default port for remote logging has been changed from 60001 to 9867; the latter lies outside most OS ephemeral port ranges.  This avoids a situation in which the connection host is localhost (the default), but there is no server listening on that port, thus permitting that port to be chosen as source for the appender connection and consequently establishing a connection to itself.  When this happens, the buffer quickly fills and blocks further logging, freezing all threads that do logging (effectively the entire domain).
pnfsmanager: Addresses compatibility issue with pre 2.6 pools

Address a compatibility issue that prevents the use of the 'rh restore' command
in pre 2.6 pools when used with a 2.6 pnfsmanager.

Target: 2.6
Require-notes: yes
Require-book: no
Patch: http://rb.dcache.org/r/5679/
Acked-by: Albert Rossi <arossi@fnal.gov>
cells: Fix login manager login limit interpretation

Addresses an issue in FTP and DCAP doors in which the login limit
was would allow n + 1 logins when n was given as a limit. Also
addresses an issue in which a limit of zero was considered to
be no limit at all. The correct value for disabling the limit is
-1.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5676/
(cherry picked from commit b83bcd5a62ca84fbef1ff5dcc0e7a8bcb9eeaff3)
info-provider: Escape percent sign in entity values

Percent signs are the only characters not allowed literally
in entity values, thus these must be escaped when exporting
the dCache configuration to an XML DTD.

The patch addresses the problem that info-provider is broken
by a recent patch adding logback formatting strings as
configuration values.

Target: trunk
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5673/
(cherry picked from commit 9b4a81ca5f627227e6fba4b66b671ffed87916b1)
poolmanager, doors: Don't retry stage requests on the same pool

Addresses a regression introduced between 1.9.12 and 2.2. The regression
means that poolmanager would not remember the previously tried stage pool
and thus could select the same pool twice in a row when retrying failed
stage requests.

The patches addresses this problem. To be effective, both pool manager
and doors have to be updated, although the patch does not break any
version compatibility.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5678/
(cherry picked from commit 4174110f6d6083908a8937e42e1c4969596e53ac)
srm: Implement SRM_FILE_BUSY response

Addresses the issue, that the SRM is ignorant to other SRM requests
it may have on the same SURL. This leads to the dreaded "Already have 1
record" error in space management, as two concurrent puts on the same
file will be allowed by the SRM, but fail in space manager. More over,
there is currently no way for the user to abort the earlier requests if
they do not have the request number.

The SRM spec says that an upload operation should cause most other
operations on that SURL to fail with SRM_FILE_BUSY. srmRm should cause
existing get and put operations on the SURL to be aborted. This patch
implements those aspects of the SRM spec.

The patch does not implement this logic for srmCopy. This will be done
in a separate patch.

Since the patch addresses a real world issue we (and probably other
sites) suffer from, I request this to be merged into 2.6. Given the
change in observable behaviour, I am reluctant to merge this into 2.2
at this point.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5677/
(cherry picked from commit 074cf4efb5e30bb8e09befad867adbb61fbfdc72)