billing: restructure to use service-oriented architecture

Combines four patches.

5415: Modify billing DAO to support p2p attribute and new request API (2)

module: core billing db/dao

Part 2.

This patch builds on http://rb.dcache.org/r/5413 to address the following:

A.  Previous versions of the billing plot generation relied on the placement of the generating cell (billing) on the same file partition as the web server.  This has made the billing plots unusable at DESY.

B.  In the current billing data tables there is no way to track migration/pool-to-pool operations as distinct from door-initiated uploads and downloads.

In support of these,

- An interface for timestamped data objects (ITimestamped) has been moved out of the billing package in core and placed in common; a second interface, IHistogramData, replaces IPlotData.  Plotting and histogram data are thus conceptually separated.

- The DAO objects have undergone small redesigning and extensions for p2p operations

- General cleanup and adjustment of main DAO implementation class

- Liquibase changelog to modify the tables

- Standalone migration script for updating data with p2p attribute values.

-----------------------------------------------------------------------------

5416: Add billing request API, handlers, and histogram implementation (3)

module: billing database and cell

Part 3.

This patch refactors the billing functionality so that it now is service-oriented.  In order to do so, we have added a general API layer for remote requests to the billing cell, providing one concrete implementation, that for histogram data.  The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction (such that plot generation can take place elsewhere, on the calling end).

New interfaces and data objects have been provided which allow for the wire transmission of an implementation-neutral data wrapper for histogram data; a service interface supporting this transfer is also implemented, with the client using the synchronous cell endpoint sendAndWait call to talk to the service, which in turn takes the request and interprets it via introspection in order invoke a method on the IBillingInfoAccess interface and return the query result to the caller.

The histogram request semantics are now the layer at which the specific JDO queries are articulated.

As stated, this implementation is actually based on a more general API for BillingRequests and their handlers, so that additional handlers could be added in the future.  The handlers are mapped by the BillingDatabase to the return type of the request object they are responsible for.

-----------------------------------------------------------------------------

5417: Refactor Jaida plot modules based on new request API (4)

module: billing plots

Part 4.

The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction.  This patch refactors plotting against the new APIs.  It is still implemented using JAIDA.  While it still resides in the dcache core module, it is actually used by the webadmin service.

We have also modified what is and what is not exposed in terms of plot configuration.  Aside from the choices for style and scale, property configuration is internal.  plotPropertiesFile path is no longer used.

The plot generation interfaces, along with the underlying JAIDA implementation, have be extended to produce the new p2p plots.

The plots generated currently use connected data points and a log scale; however, DESY has plots which use an bar/outline style. We would like to allow the installation or the user to display the plots using a few simple options; hence, we have added options and default properties for them:

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  (These options are actually made available via user-facing choices on the new billing plots page; see the final patch in this series).

-----------------------------------------------------------------------------

5578: Changes to webadmin to accommodate new billing SOA; no plot rendering options (5)
module: webadmin billing

Part 5.  Changes to webadmin in connection with the reworked billing infrastructure.

Version of 5418 in which the plot rendering options are not exposed to the user.

Conversion to implementation-specific histograms and plots (in the current case, using the org.freehep packages) now takes place on the receiving end of the call.  The JAIDA-based classes are included in the core module, but are now utilized inside of the webadmin container.

This is accomplished by adding a service abstraction and implementation to the controller package of webadmin; this service is used by the billing page.

Refresh can be achieved (a) by forcing it with the refresh button, (b) leaving the window open Ajax auto-refresh is added to the form, run once every refresh interval) and/or (c), a daemon thread in the service which runs at the refresh interval.

Property configuration is internal.  plotPropertiesFile path is no longer used.

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  Once the default is selected, this is the form in which the service serves up the plots to everyone.

-----------------------------------------------------------------------------

Testing:

For the redesign, a fully redeployed RPM was tested with the billing cell running both inside and outside the httpd domain.  Tests for the following conditions:

    - reload with no plots
    - reload with unexpired plots
    - reload with expired plots
    - force refresh

were run in Firefox, Safari and Chrome.

Also tested were the default settings for the style and scaling, and switching between those options.

For p2p, multiple migrations done from pool to pool through admin interface and logs/database/plots checked for consistency.

-----------------------------------------------------------------------------

Patch: http://rb.dcache.org/r/5415
Patch: http://rb.dcache.org/r/5416
Patch: http://rb.dcache.org/r/5417
Patch: http://rb.dcache.org/r/5578
Acked-by: Gerd
Target: master
Require-book: yes
Require-notes: yes
Request: 2.6

RELEASE NOTES:
BOOK:
Draw attention to the possibility of adding a clause in the accounting logs (format for mover messages): '[p2p=true/false]'.  The database schema for the billinginfo table has been modified to include this boolean 'p2p' indicating a pool-to-pool transfer.  An sql script for migrating old data (an optional procedure to make the billing table consistent which will require several hours of downtime) is included (/usr/share/dcache/migration/migrate_to_billing_with_p2p.sql).

The billing cell now is independent of the domain where the plots are rendered (the webadmin service); there is no longer a requirement to run billing on the same file partition as the httpd domain.  note that "billingToDb" needs to be visible only to the billing service, and "generatePlots" only to the httpd service.  Note that plotPropertiesFile is obsolete and no longer used. There several new properties for controlling plot types.

webadmin.billing-plot.refresh-threshold=5

webadmin.billing-plot.height=800

webadmin.billing-plot.width=1200

(one-of?CONNECTED|OUTLINE|FILLED)webadmin.billing-plot.default-style=CONNECTED

(one-of?lin|log)webadmin.billing-plot.default-scale=log
Merge remote branch 'main/2.6' into feature/2.6/billing_soa

Conflicts:
	skel/share/defaults/dcache.properties
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
merged: http://rb.dcache.org/r/5641
merged: http://rb.dcache.org/r/5646
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
Merged master patch http://rb.dcache.org/r/5670
dcache: (remote logging) change default server port to be outside ephemeral port range

module: dcache (alarms)

This patch is an added precaution in connection with http://rb.dcache.org/r/5664.  It was discovered that the default port suggested by ch.qos.logback for the SocketAppender and server, 60001, is in fact not a good choice because it lies within the ephemeral port range for some OS's.  When there is no endpoint listening (I believe this only occurs when the server is supposed to be running local to the client domain), that port can be chosen as source for establishing a connection, such that the appender then ends up connecting to itself.  If events are actually sent over this connection, it quickly fills up the buffer and blocks.  This causes all threads which do logging to block, effectively freezing the system.

The above-mentioned patch sets remote logging by default to off, so that events are not sent unless it is explicitly activated.  The self-connected appender, however, is still potentially present (though at the 10 second reconnect timeout -- which is what the logback.xml is configured for -- it will take a while for it to find the 60001 port).

Setting the default port outside most ephemeral port ranges will guarantee that this connect-back does not happen.

Testing done:

Using a small test class with the remote logger reconnect delay set to 1 ms, and no server endpoint running, port 60001 was first chosen.  After 30 seconds,

[root@otfrid dcache]# netstat -anp | grep 60001
tcp        4      0 127.0.0.1:60001             127.0.0.1:60001             ESTABLISHED 6051/java

was seen.  The test waits 60 seconds, then begins to log continuously. The system freezes after 5446 events:

6001 25 Jun 2013 08:03:24 [] error message 1
...
6001 25 Jun 2013 08:03:24 [] error message 5446

Rerunning with port set to 9876 shows that after 10 minutes and more than 17M events, no connection is established and hence no freeze.

Target: 2.6
Require-notes: yes
Require-book: no
Acked-by: Gerd
Bug: http://rt.dcache.org/Ticket/Display.html?id=7856
Merge-req:

RELEASE NOTES:

The default port for remote logging has been changed from 60001 to 9867; the latter lies outside most OS ephemeral port ranges.  This avoids a situation in which the connection host is localhost (the default), but there is no server listening on that port, thus permitting that port to be chosen as source for the appender connection and consequently establishing a connection to itself.  When this happens, the buffer quickly fills and blocks further logging, freezing all threads that do logging (effectively the entire domain).
pnfsmanager: Addresses compatibility issue with pre 2.6 pools

Address a compatibility issue that prevents the use of the 'rh restore' command
in pre 2.6 pools when used with a 2.6 pnfsmanager.

Target: 2.6
Require-notes: yes
Require-book: no
Patch: http://rb.dcache.org/r/5679/
Acked-by: Albert Rossi <arossi@fnal.gov>
cells: Fix login manager login limit interpretation

Addresses an issue in FTP and DCAP doors in which the login limit
was would allow n + 1 logins when n was given as a limit. Also
addresses an issue in which a limit of zero was considered to
be no limit at all. The correct value for disabling the limit is
-1.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5676/
(cherry picked from commit b83bcd5a62ca84fbef1ff5dcc0e7a8bcb9eeaff3)
info-provider: Escape percent sign in entity values

Percent signs are the only characters not allowed literally
in entity values, thus these must be escaped when exporting
the dCache configuration to an XML DTD.

The patch addresses the problem that info-provider is broken
by a recent patch adding logback formatting strings as
configuration values.

Target: trunk
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Christian Bernardt <christian.bernardt@desy.de>
Patch: http://rb.dcache.org/r/5673/
(cherry picked from commit 9b4a81ca5f627227e6fba4b66b671ffed87916b1)
poolmanager, doors: Don't retry stage requests on the same pool

Addresses a regression introduced between 1.9.12 and 2.2. The regression
means that poolmanager would not remember the previously tried stage pool
and thus could select the same pool twice in a row when retrying failed
stage requests.

The patches addresses this problem. To be effective, both pool manager
and doors have to be updated, although the patch does not break any
version compatibility.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: yes
Require-book: no
Acked-by: Albert Rossi <arossi@fnal.gov>
Patch: http://rb.dcache.org/r/5678/
(cherry picked from commit 4174110f6d6083908a8937e42e1c4969596e53ac)
srm: Implement SRM_FILE_BUSY response

Addresses the issue, that the SRM is ignorant to other SRM requests
it may have on the same SURL. This leads to the dreaded "Already have 1
record" error in space management, as two concurrent puts on the same
file will be allowed by the SRM, but fail in space manager. More over,
there is currently no way for the user to abort the earlier requests if
they do not have the request number.

The SRM spec says that an upload operation should cause most other
operations on that SURL to fail with SRM_FILE_BUSY. srmRm should cause
existing get and put operations on the SURL to be aborted. This patch
implements those aspects of the SRM spec.

The patch does not implement this logic for srmCopy. This will be done
in a separate patch.

Since the patch addresses a real world issue we (and probably other
sites) suffer from, I request this to be merged into 2.6. Given the
change in observable behaviour, I am reluctant to merge this into 2.2
at this point.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5677/
(cherry picked from commit 074cf4efb5e30bb8e09befad867adbb61fbfdc72)
dcache: alarms: Extend log level capture to all levels; fix server-side logging

module: dcache, package alarms

Because of issues that were not working with the Turbo Filter, the Alarm definition was implemented to accept only messages at WARN or ERROR levels.  It is preferable to allow an ALARM to be defined against any logging level.

I have retested the Turbo Filter and it is now working as advertised.  This patch

- moves the alarms.remote-logging.level property to the level element of the remote appender in the TurboFilter, where it belongs, and eliminates the extra filter on the appender itself;
- makes the necessary modifications to allow ALARMS to be captured at any level, even though ERROR and WARN will undoubtedly be the most common (and even though to capture those below the WARN level one risks a considerable amount of congestion on the alarm server; in such a case it is advisable not to use the XML store).

The logback-server.xml has also been adjusted so that:

- the log file messages are fuller and resemble the optional MAIL message;
- events from the server's own domain are excluded from the store and the history and are placed in the server.log

Testing:

On testbed.  Tested by varying the levels of alarms.remote-logging.level (set globally in the dcache.conf file) and observing the results in the history.log, server.log and on the webadmin page.  Also added an arbitrary ALARM defined at INFO level and observed it appear on the page when the remote logging level was low enough.

Target: master
Patch: http://rb.dcache.org/r/5522
Require-book: yes
Require-notes: yes
Acked-by: Gerd
Request: 2.6

RELEASE NOTES:
BOOK:

Alarms can now be defined at all logging levels.  However, the volume of traffic sent to the server (where the logging message is captured and converted into a potential alarm) increases dramatically below the WARN level.  If this kind of alarm is desired, it is advisable to run the RDBMS version of the store, as the XML store fills up to scores of pages very quickly and searching becomes slow.  Logging messages from the logging server and the domain in which it runs are not sent to the store, but are written to the server.log.

Conflicts:
	skel/var/alarms/logback-server.xml
dcache-webadmin: avoid rrd4j timestep error statement

module: dcache-webadmin (pool queue plots)

Under certain restart conditions, the rrd4j utility will report that the call to write is doing so with a timestamp which is less than 1 second since the last update.  While this error is not fatal, it is annoying to see in the log.

The extra check simply avoids writing the data under these conditions and waits till the next iteration.

Target: master@39b9580e1820d1599399db78665ad37c386c7018
Patch: http://rb.dcache.org/r/5680
Require-notes: yes
Require-book: no
Request: 2.6
Acked-by: Gerd

Testing Done:

Without the patch, rapid multiple restarts of the webadmin service/httpd domain will probably produce a trace similar to this:

26 Jun 2013 09:32:47 (System) [] problem running collectPoolSeclectionUnit
java.lang.IllegalArgumentException: Bad sample time: 1372257000. Last update time was 1372257000, at least one second step is required
        at org.rrd4j.core.RrdDb.store(RrdDb.java:553) ~[rrd4j-2.1.1.jar:na]
        at org.rrd4j.core.Sample.update(Sample.java:197) ~[rrd4j-2.1.1.jar:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.storeToRRD(RrdPoolInfoAgent.java:331) ~[classes/:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.processPools(RrdPoolInfoAgent.java:291) ~[classes/:na]
        at org.dcache.webadmin.model.dataaccess.util.RrdPoolInfoAgent.run(RrdPoolInfoAgent.java:138) ~[classes/:na]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0]

With the patch, the log remains clean during multiple successive attempts to restart.

RELEASE NOTES:  The cause of the stack trace occasionally seen in the webadmin log file (on restart of the service) has been addressed.
dcache: (billing) Fix incorrect MoverData field name

Regression introduced by last large multi-patch billing update.

The MoverData field 'p2p' was renamed to 'isPtp', following suggestion that Java convention be followed.  However, the liquibase and package.jdo files continued to assume the database table attribute was 'p2p'.  Hence when the data object gets dehydrated by DataNucleus, the insert statement fails.

This patch reverts the Java field name to 'p2p'.

Testing done:

Without the patch, writes to the billing database will produce the following error:

02 jul 2013 05:43:03 (billing) [] Insert of object "org.dcache.services.billing.db.data.MoverData@6e19f13" using statement "INSERT INTO "billinginfo"
("transfersize","fullsize","cellname","datestamp","protocol","action","errormessage","initiator","transaction","storageclass","errorcode","client","isp2p","pnfsid","isnew","connectiontime")
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : user lacks privilege or object not found: isp2p
02 jul 2013 05:43:03 (billing) [] could not not commit; some billing
data may be lost

With the patch, successful srmcp produced the correct entry in billinginfo.

Note:  this has been tested on a regular instance of dcache, not using system-test.

Target: master@60c28664059ea20aea204ef9dd0d35c89b124caf
Patch: http://rb.dcache.org/r/5692/
Require-book: no
Require-notes: yes
Request: 2.6
Acked-by: Gerd

RELEASE NOTES:  Fixes bug preventing inserts into the billinginfo table.

Ticket: #number Acked-by: Target: trunk Require-book: no Require-notes: no Request: Bug: Merge-req:
pool: Fix memory leak in xrootd and httpd movers

Addresses a slow memory leak that occurs whenever more than one httpd
or more than one xrootd transfer is started on a pool. Also reduces
http/xrootd server component startup overhead.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5695/
(cherry picked from commit 7679d8ddd80872b9301869846f71f413976281c5)
cells: Fix race conditions in cell lookup

When cells are killed, they are moved from one map to another map. This operation
is not atomic and methods that look in both places need to be synchronized.

This addresses a null pointer exception experienced in the DCAP door:

java.lang.NullPointerException: null
        at diskCacheV111.doors.DCapDoorInterpreterV3.ac_get_door_info(DCapDoorInterpreterV3.java:733) ~[dcache-dcap-2.2.13.jar:2.2.13]
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_19]
        at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_19]
        at dmg.util.CommandInterpreter.execute(CommandInterpreter.java:609) ~[cells-2.2.13.jar:2.2.13]
        at dmg.util.CommandInterpreter.command(CommandInterpreter.java:486) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellAdapter.executeLocalCommand(CellAdapter.java:1020) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellAdapter.messageArrived(CellAdapter.java:940) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellNucleus$DeliverMessageTask.innerRun(CellNucleus.java:1051) ~[cells-2.2.13.jar:2.2.13]
        at dmg.cells.nucleus.CellNucleus$AbstractNucleusTask.run(CellNucleus.java:949) ~[cells-2.2.13.jar:2.2.13]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_19]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_19]
        at dmg.cells.nucleus.CellNucleus$1.run(CellNucleus.java:654) [cells-2.2.13.jar:2.2.13]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_19]

Target: trunk
Request: 2.6
Request: 2.2
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7896
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5700/
(cherry picked from commit 45850f5d50d867772d91b8526d9b7210ae9aaa55)
cells: Fix deadlock

Addresses a regression introduced in 45850f5d50d867772d91b8526d9b7210ae9aaa55. The
regression introduces a deadlock caused by low-level class loading logic in AspectJ
using the logging system, and our logging system looks up cells through getCell(),
a method made synchronized by the above commit.

The solution adopted is to not have two cell maps. Rather we have a set of killed
cells to track the fact that a cell was killed.

Target: trunk
Request: 2.6
Request: 2.2
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5718/
(cherry picked from commit d4fd3244a9770f811f92bcf429902029f30442fc)

Conflicts:
	modules/cells/src/main/java/dmg/cells/nucleus/CellGlue.java
pool: Add missing attributes in replicateOnArrival request

Addresses an issue in the pool's replicateOnArrival feature. If enabled it
failed with an error stating that required attributes where missing.

Target: trunk
Request: 2.6
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5715/
(cherry picked from commit a4188cfc4e89e2c5310fef52fc16f120d9c915db)
Merge branch '2.6' into fix/2.6/rb5522

Conflicts:
	skel/share/defaults/alarms.properties