dcache: alarms: Extend log level capture to all levels; fix server-side logging

module: dcache, package alarms

Because of issues that were not working with the Turbo Filter, the Alarm definition was implemented to accept only messages at WARN or ERROR levels.  It is preferable to allow an ALARM to be defined against any logging level.

I have retested the Turbo Filter and it is now working as advertised.  This patch

- moves the alarms.remote-logging.level property to the level element of the remote appender in the TurboFilter, where it belongs, and eliminates the extra filter on the appender itself;
- makes the necessary modifications to allow ALARMS to be captured at any level, even though ERROR and WARN will undoubtedly be the most common (and even though to capture those below the WARN level one risks a considerable amount of congestion on the alarm server; in such a case it is advisable not to use the XML store).

The logback-server.xml has also been adjusted so that:

- the log file messages are fuller and resemble the optional MAIL message;
- events from the server's own domain are excluded from the store and the history and are placed in the server.log

Testing:

On testbed.  Tested by varying the levels of alarms.remote-logging.level (set globally in the dcache.conf file) and observing the results in the history.log, server.log and on the webadmin page.  Also added an arbitrary ALARM defined at INFO level and observed it appear on the page when the remote logging level was low enough.

Target: master
Patch: http://rb.dcache.org/r/5522
Require-book: yes
Require-notes: yes
Acked-by: Gerd
Request: 2.6

RELEASE NOTES:
BOOK:

Alarms can now be defined at all logging levels.  However, the volume of traffic sent to the server (where the logging message is captured and converted into a potential alarm) increases dramatically below the WARN level.  If this kind of alarm is desired, it is advisable to run the RDBMS version of the store, as the XML store fills up to scores of pages very quickly and searching becomes slow.  Logging messages from the logging server and the domain in which it runs are not sent to the store, but are written to the server.log.
dcache: alarm service -- change default db to rdbms

module: dcache

Currently the default database plugin is the datanucleus xml adapter.  This was perfectly fine when the alarm server was originally conceived of, because the client end was only sending alarm events, so the load was small.  Subsequently, we re-implemented the service to be a general log message receiver, with the level of events both sent and filtered on the server end being totally configurable.  Hence it is now possible to fill up the xml store very quickly if the event level is set below warn.  The slowdown for all queries (since it does not seem the XML plugin is capable of optimizations) is quite noticeable, and the page takes a bit to load.

It thus no longer makes sense to set the default database for alarms to xml.  This option is still available for those who do not intend to send anything but error or warn level events to the service, but our out-of-the-box recommendation should be rdbms (postgres), even if it requires the one additional setup step of creating the db (as one does for chimera, dcache and billing).

Target: master
Patch: http://rb.dcache.org/r/5553/
Require-notes: yes
Require-book: yes
Request: 2.6
Acked-by: Dmitry

RELEASE NOTES:
BOOK:

The default values for alarms.store.db.type has been changed from "xml" to "rdbms".  This is done so as to accommodate the likelihood that warn messages (the default remote logging level) will accumulate but will not degrade the performance of the alarm queries or the loading of the alarms table.  The xml option is still available at the discretion of the admin.  Remember that the alarms database must be created as you do for billing, dcache, etc., before booting the alarmserver domain.  Concomitantly, the default value for webadmin.alarm.cleaner.enabled has been changed from true to false, as the daemon which cleans out old entries is not essential when using rdbms.
dcache: billing -- fix error in liquibase varchar changeset

module: dcache

The liquibase changeset 4.1.7 was discovered to contain an off-by-one error in the string used to test the condition for changing the varchar length in the billing tables (the actual length of the string was 256, not 257). As a result, 4.1.7. 4.1.8 and 4.1.9 never will run, and all billing installations created by liquibase (which hardcodes the default to 256) remained as such.

The reason for the requested change was that the path and errormessage attributes of certain tables can well exceed 256.

The new changesets correct the error.

Target: master
Patch: http://rb.dcache.org/r/5555
Require-book: no
Require-notes: yes
Request: 2.2
Request: 2.6
Acked-by: Dmitry

RELEASE NOTES:  Fixes incorrect sizes on billing table varchar attributes pnfsid, path and errormessage.  The problem arises with the billing databases that were generated by Liquibase and not by hand.
This is because Liquibase hardcodes the varchar default to 256, where postgresql leaves it undefined (unlimited).  A previous patch to correct this was discovered to contain a bug which prevented it
from running.

TESTING:

After liquibase, tables for pnfsid, path and errormessage now correct size:

billing=> \d billinginfo
              Table "public.billinginfo"
     Column     |           Type           | Modifiers
----------------+--------------------------+-----------
 client         | character varying(256)   |
 initiator      | character varying(256)   |
 isnew          | boolean                  |
 protocol       | character varying(256)   |
 transfersize   | bigint                   |
 fullsize       | bigint                   |
 storageclass   | character varying(256)   |
 connectiontime | bigint                   |
 action         | character varying(256)   |
 cellname       | character varying(256)   |
 datestamp      | timestamp with time zone |
 errorcode      | integer                  |
 errormessage   | character varying(8000)  |
 pnfsid         | character varying(36)    |
 transaction    | character varying(256)   |
 p2p            | boolean                  |
Indexes:
    "billinginfo_client_idx" btree (client)
    "billinginfo_date_idx" btree (datestamp)
    "billinginfo_initiator_idx" btree (initiator)
    "billinginfo_isnew_idx" btree (isnew)
    "billinginfo_p2p_idx" btree (p2p)
    "billinginfo_pnfsid_idx" btree (pnfsid)
    "billinginfo_storageclass_idx" btree (storageclass)
    "billinginfo_transaction_idx" btree (transaction)
Triggers:
    tgr_update_billinginfo_p2p_daily AFTER INSERT ON billinginfo FOR EACH ROW EXECUTE PROCEDURE f_update_billinginfo_p2p_daily()
    tgr_update_billinginfo_rd_daily AFTER INSERT ON billinginfo FOR EACH ROW EXECUTE PROCEDURE f_update_billinginfo_rd_daily()
    tgr_update_billinginfo_tm_daily AFTER INSERT ON billinginfo FOR EACH ROW EXECUTE PROCEDURE f_update_billinginfo_tm_daily()
    tgr_update_billinginfo_wr_daily AFTER INSERT ON billinginfo FOR EACH ROW EXECUTE PROCEDURE f_update_billinginfo_wr_daily()

billing=> \d doorinfo
                Table "public.doorinfo"
     Column     |           Type           | Modifiers
----------------+--------------------------+-----------
 client         | character varying(256)   |
 mappedgid      | integer                  |
 mappeduid      | integer                  |
 owner          | character varying(256)   |
 path           | character varying(8000)  |
 queuedtime     | bigint                   |
 connectiontime | bigint                   |
 action         | character varying(256)   |
 cellname       | character varying(256)   |
 datestamp      | timestamp with time zone |
 errorcode      | integer                  |
 errormessage   | character varying(8000)  |
 pnfsid         | character varying(36)    |
 transaction    | character varying(256)   |
Indexes:
    "doorinfo_date_idx" btree (datestamp)
    "doorinfo_owner_idx" btree (owner)
    "doorinfo_pnfsid_idx" btree (pnfsid)
    "doorinfo_transaction_idx" btree (transaction)

billing=> \d storageinfo
              Table "public.storageinfo"
     Column     |           Type           | Modifiers
----------------+--------------------------+-----------
 queuedtime     | bigint                   |
 fullsize       | bigint                   |
 storageclass   | character varying(256)   |
 connectiontime | bigint                   |
 action         | character varying(256)   |
 cellname       | character varying(256)   |
 datestamp      | timestamp with time zone |
 errorcode      | integer                  |
 errormessage   | character varying(8000)  |
 pnfsid         | character varying(36)    |
 transaction    | character varying(256)   |
Indexes:
    "storageinfo_action_idx" btree (action)
    "storageinfo_date_idx" btree (datestamp)
    "storageinfo_pnfsid_idx" btree (pnfsid)
    "storageinfo_storageclass_idx" btree (storageclass)
    "storageinfo_transaction_idx" btree (transaction)
Triggers:
    tgr_update_storageinfo_rd_daily AFTER INSERT ON storageinfo FOR EACH ROW EXECUTE PROCEDURE f_update_storageinfo_rd_daily()
    tgr_update_storageinfo_wr_daily AFTER INSERT ON storageinfo FOR EACH ROW EXECUTE PROCEDURE f_update_storageinfo_wr_daily()

billing=> \d hitinfo
               Table "public.hitinfo"
    Column    |           Type           | Modifiers
--------------+--------------------------+-----------
 filecached   | boolean                  |
 action       | character varying(256)   |
 cellname     | character varying(256)   |
 datestamp    | timestamp with time zone |
 errorcode    | integer                  |
 errormessage | character varying(8000)  |
 pnfsid       | character varying(36)    |
 transaction  | character varying(256)   |
Indexes:
    "hitinfo_date_idx" btree (datestamp)
    "hitinfo_pnfsid_idx" btree (pnfsid)
    "hitinfo_transaction_idx" btree (transaction)
Triggers:
    tgr_update_hitinfo_daily AFTER INSERT ON hitinfo FOR EACH ROW EXECUTE PROCEDURE f_update_hitinfo_daily()
Revert "dcache: billing -- fix error in liquibase varchar changeset"

This reverts commit db5ddfd6566edcbffa0829f16314bc9c2f0a3da8.

The select query needs an added where clause to make it run faster.
pool: Clean up StorageClassContainer and StorageClassInfo

- Addresses catch all exception handlers

- Removes redundant locking in StorageClassContainer

- Removes unused fields

- Addresses synchronization issues

- Rewrites shell commands as annotated commands (better help output,
  better tab completion in the future).

- Drops deprecated 'define class' command (needs to be mentioned
  in release notes).

- Use thread pool rather creating new threads on every callback

- Avoid unnecessary collection copies

Target: trunk
Require-book: no
Require-notes: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5468/
pool: Clean up HSM storage handler

Addresses code smell in HsmStorageHandler2 class.

- Prefer setter injection over constructor injection

- Prefer annotated CLI command

- Document CLI commands

- Avoid raw types

- Avoid dead code

- Consistent source code formatting

- Prefer logging patterns over string concatenation

- Embed CLI command in HsmStorageHandler2. Having them in a separate
  class led to an overly complex interface for HsmStorageHandler2 and
  led to race conditions and inefficient code.

This is the first of several patches for HsmStorageHandler2.

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5478/
pool: Clean up job scheduler used for HSM

Addresses code smell in SimpleJobScheduler. The output format of
'rh jobs ls' and 'st jobs ls' is changed as the priority, client
and client ID are no longer shown - these fields were not used
and always contained the same value for HSM jobs.

JobScheduler and its implementation is only used by HSM jobs
and thus much of its functionality was unused. The patch
removes a lot of dead code.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5480/
pool: Replace redundant st/rh jobs remove/kill commands with st/rh kill

Addresses the functionality overlap between the st jobs remove/
rh jobs remove and st jobs kill/rh jobs kill. These commands are
replaced with the new st kill and rh kill commands, respectively.

Addresses a race condition between killing an HSM job and the
start of the HSM script.

Resulting dead code is removed from SimpleJobScheduler. HSM script
invocation is modified to translate return code 143 (sigterm) to
a TimeoutCacheException.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5482/
pool: Remove redudant rh/st jobs ls commands

Addresses duplicated functionality in the commands rh/st jobs ls
and rh/st ls. Both commands listed the restores/store requests on
the pool, although with different attributes. The additional
attributes have been added to rh/st ls and rh/st jobs ls has been
removed.

Target: trunk
Require-notes: yes
Require-book: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5483/
pool: Fix race conditions in HSM management commands

Addresses race conditions in HSM management commands. These
race conditions could lead to corrupted HSM definitions and
infinite loops during flush and stage.

It is unlikely that one would hit any of these races in the
real world. For this reason I am not requesting a merge to
stable branches.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/5486/
pool: Minor refactoring of HsmStorageHandler2

- Don't expose internal queue to other components.

- Make internal stuff private.

- Move utility methods to the correct class.

- Remove unnecessary synchronization

Target: trunk
Require-book: no
Require-notes: no
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5491/
Minor bug fix upgrades for third party dependencies

Addresses the issue that several third party libraries used by dCache
have bug fix releases newer than the version we use.

I would have liked to get these committed before 2.6.0 got released,
but didn't have the time to do it.

Target: master
Request: 2.6
Require-book: no
Require-notes: yes
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5549/
nfsv4: initialise CDC for better logging

Acked-by: Paul Millar
Target: trunk, 2.6
Require-book: no
Require-notes: no