billing: restructure to use service-oriented architecture

Combines four patches.

5415: Modify billing DAO to support p2p attribute and new request API (2)

module: core billing db/dao

Part 2.

This patch builds on http://rb.dcache.org/r/5413 to address the following:

A.  Previous versions of the billing plot generation relied on the placement of the generating cell (billing) on the same file partition as the web server.  This has made the billing plots unusable at DESY.

B.  In the current billing data tables there is no way to track migration/pool-to-pool operations as distinct from door-initiated uploads and downloads.

In support of these,

- An interface for timestamped data objects (ITimestamped) has been moved out of the billing package in core and placed in common; a second interface, IHistogramData, replaces IPlotData.  Plotting and histogram data are thus conceptually separated.

- The DAO objects have undergone small redesigning and extensions for p2p operations

- General cleanup and adjustment of main DAO implementation class

- Liquibase changelog to modify the tables

- Standalone migration script for updating data with p2p attribute values.

-----------------------------------------------------------------------------

5416: Add billing request API, handlers, and histogram implementation (3)

module: billing database and cell

Part 3.

This patch refactors the billing functionality so that it now is service-oriented.  In order to do so, we have added a general API layer for remote requests to the billing cell, providing one concrete implementation, that for histogram data.  The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction (such that plot generation can take place elsewhere, on the calling end).

New interfaces and data objects have been provided which allow for the wire transmission of an implementation-neutral data wrapper for histogram data; a service interface supporting this transfer is also implemented, with the client using the synchronous cell endpoint sendAndWait call to talk to the service, which in turn takes the request and interprets it via introspection in order invoke a method on the IBillingInfoAccess interface and return the query result to the caller.

The histogram request semantics are now the layer at which the specific JDO queries are articulated.

As stated, this implementation is actually based on a more general API for BillingRequests and their handlers, so that additional handlers could be added in the future.  The handlers are mapped by the BillingDatabase to the return type of the request object they are responsible for.

-----------------------------------------------------------------------------

5417: Refactor Jaida plot modules based on new request API (4)

module: billing plots

Part 4.

The billing package in core has been restructured so that the histogram abstraction is distinct from the plot abstraction.  This patch refactors plotting against the new APIs.  It is still implemented using JAIDA.  While it still resides in the dcache core module, it is actually used by the webadmin service.

We have also modified what is and what is not exposed in terms of plot configuration.  Aside from the choices for style and scale, property configuration is internal.  plotPropertiesFile path is no longer used.

The plot generation interfaces, along with the underlying JAIDA implementation, have be extended to produce the new p2p plots.

The plots generated currently use connected data points and a log scale; however, DESY has plots which use an bar/outline style. We would like to allow the installation or the user to display the plots using a few simple options; hence, we have added options and default properties for them:

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  (These options are actually made available via user-facing choices on the new billing plots page; see the final patch in this series).

-----------------------------------------------------------------------------

5578: Changes to webadmin to accommodate new billing SOA; no plot rendering options (5)
module: webadmin billing

Part 5.  Changes to webadmin in connection with the reworked billing infrastructure.

Version of 5418 in which the plot rendering options are not exposed to the user.

Conversion to implementation-specific histograms and plots (in the current case, using the org.freehep packages) now takes place on the receiving end of the call.  The JAIDA-based classes are included in the core module, but are now utilized inside of the webadmin container.

This is accomplished by adding a service abstraction and implementation to the controller package of webadmin; this service is used by the billing page.

Refresh can be achieved (a) by forcing it with the refresh button, (b) leaving the window open Ajax auto-refresh is added to the form, run once every refresh interval) and/or (c), a daemon thread in the service which runs at the refresh interval.

Property configuration is internal.  plotPropertiesFile path is no longer used.

The plot style can be CONNECTED, OUTLINE or FILLED.  If OUTLINE, the first histogram appears as a filled bar and all others overlaid are outlines only.  FILLED, on the other hand, overlays histograms opaquely over each other.  The default can be set via the property billing.plot.default-style.  Scale can be "lin" (linear) or "log" (logarithmic); default property set via billing.plot.default-scale.  Once the default is selected, this is the form in which the service serves up the plots to everyone.

-----------------------------------------------------------------------------

Testing:

For the redesign, a fully redeployed RPM was tested with the billing cell running both inside and outside the httpd domain.  Tests for the following conditions:

    - reload with no plots
    - reload with unexpired plots
    - reload with expired plots
    - force refresh

were run in Firefox, Safari and Chrome.

Also tested were the default settings for the style and scaling, and switching between those options.

For p2p, multiple migrations done from pool to pool through admin interface and logs/database/plots checked for consistency.

-----------------------------------------------------------------------------

Patch: http://rb.dcache.org/r/5415
Patch: http://rb.dcache.org/r/5416
Patch: http://rb.dcache.org/r/5417
Patch: http://rb.dcache.org/r/5578
Acked-by: Gerd
Target: master
Require-book: yes
Require-notes: yes
Request: 2.6

RELEASE NOTES:
BOOK:
Draw attention to the possibility of adding a clause in the accounting logs (format for mover messages): '[p2p=true/false]'.  The database schema for the billinginfo table has been modified to include this boolean 'p2p' indicating a pool-to-pool transfer.  An sql script for migrating old data (an optional procedure to make the billing table consistent which will require several hours of downtime) is included (/usr/share/dcache/migration/migrate_to_billing_with_p2p.sql).

The billing cell now is independent of the domain where the plots are rendered (the webadmin service); there is no longer a requirement to run billing on the same file partition as the httpd domain.  note that "billingToDb" needs to be visible only to the billing service, and "generatePlots" only to the httpd service.  Note that plotPropertiesFile is obsolete and no longer used. There several new properties for controlling plot types.

webadmin.billing-plot.refresh-threshold=5

webadmin.billing-plot.height=800

webadmin.billing-plot.width=1200

(one-of?CONNECTED|OUTLINE|FILLED)webadmin.billing-plot.default-style=CONNECTED

(one-of?lin|log)webadmin.billing-plot.default-scale=log
Merge remote branch 'main/2.6' into feature/2.6/billing_soa

Conflicts:
	skel/share/defaults/dcache.properties
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
merged: http://rb.dcache.org/r/5641
merged: http://rb.dcache.org/r/5646
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
Merged master patch http://rb.dcache.org/r/5670
dcache: (billing) Fix incorrect MoverData field name

Regression introduced by last large multi-patch billing update.

The MoverData field 'p2p' was renamed to 'isPtp', following suggestion that Java convention be followed.  However, the liquibase and package.jdo files continued to assume the database table attribute was 'p2p'.  Hence when the data object gets dehydrated by DataNucleus, the insert statement fails.

This patch reverts the Java field name to 'p2p'.

Testing done:

Without the patch, writes to the billing database will produce the following error:

02 jul 2013 05:43:03 (billing) [] Insert of object "org.dcache.services.billing.db.data.MoverData@6e19f13" using statement "INSERT INTO "billinginfo"
("transfersize","fullsize","cellname","datestamp","protocol","action","errormessage","initiator","transaction","storageclass","errorcode","client","isp2p","pnfsid","isnew","connectiontime")
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" failed : user lacks privilege or object not found: isp2p
02 jul 2013 05:43:03 (billing) [] could not not commit; some billing
data may be lost

With the patch, successful srmcp produced the correct entry in billinginfo.

Note:  this has been tested on a regular instance of dcache, not using system-test.

Target: master@60c28664059ea20aea204ef9dd0d35c89b124caf
Patch: http://rb.dcache.org/r/5692/
Require-book: no
Require-notes: yes
Request: 2.6
Acked-by: Gerd

RELEASE NOTES:  Fixes bug preventing inserts into the billinginfo table.

Ticket: #number Acked-by: Target: trunk Require-book: no Require-notes: no Request: Bug: Merge-req:
Merge remote branch 'main/2.6' into feature/2.6/billing_soa
Removed the old (superfluous) loginBroker=LoginBroker definition for dcache.properties.
Merge pull request #91 from alrossi/feature/2.6/billing_soa

billing: restructure to use service-oriented architecture