Deprecate a few old properties

The patch fixes a few inconsistencies spotted by Antje.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: no
Acked-by: Antje Petersen <antje.petersen@desy.de>
Patch: http://rb.dcache.org/r/5963/
Replace db.schema.auto with dcache.db.schema.auto

Thanks to Antje for spotting this inconsistency.

Target: trunk
Request: 2.7
Require-notes: yes
Require-book: yes
Acked-by: Antje Petersen <antje.petersen@desy.de>
Patch: http://rb.dcache.org/r/5972/
admin: fail commands when user supplies an option that is unknown

The current behaviour for both annotated and ac_ commands is that
any options that are unknown to the command are simply ignored.

This behaviour can be dangerous; for example, the migration
module's move and copy commands accept a 'pnfsid' option to
limit the scope of the operation.  Multiple PNFS-IDs may be
specified as a comma-separated list; however, this leads to
the potential for invoking a move with the option:

     -pnfsids=<id1>,<id2>,...

The migration module will silently ignore the unknown 'pnfsids'
option and move or copy all files on the pool.

More generally, if there is an unknown option then the user is
intending some behaviour that the command will not honour:
irrespective of how dangerous this is, it can be frustrating
to diagnose why an apparently correct invocation isn't doing
what is intended.

This patch fixes this problem for annotated commands only.
This is sufficient to fix the specific issue mentioned in the
ticket.  Solving this for non-annotated commands might be
possible, but I believe only with considerable effort.  Such
effort would be better spent moving those commands to use the
new annotation-based command format.

Target: trunk
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=7991
Patch: http://rb.dcache.org/r/5957/
Acked-by: Albert Rossi
webadmin: wait untill billing cell is ready

StandardBillingService should wait untill billing cell is ready

Ticket: #7981
Acked-by: Albert Rossi
Acked-By: Paul Millar
Target: 2.6, 2.7, master

Require-book: no
Require-notes: no
Merge remote-tracking branch 'upstream/master'
Change ssh admin properties documentation, historyPath
srm: Fix regression in srmCopy

CopyRequest and CopyFileRequest are both processed by the
same SRM scheduler. Thus the scheduler must be configured
for a common base class of these two jobs.

Target: trunk
Request: 2.7
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5996/
srm: Do not use SQLException unless its about JDBC

SQLException is the main exception used by JDBC code. The SRM in many
places declares SQLException in code completely unrelated to JDBC, in
code that doesn't even throw SQLException, or uses it to propagate
non-JDBC errors.

This patch addresses this problem, mostly by removing the declaration
of SQLException in the throws clause of many methods. In some cases
SQLException is replaced by another exception, eg IOException or
SRMException.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/5953/
srm: Use Spring JDBC support

Spring has extensive support for simplifying JDBC code. This patch updates
the srm to make use of Spring's JDBC support.

At the core of the patch is Spring's JdbcTemplate that provides methods
for common JDBC patterns. Given that the existing code was Connection
oriented, most uses of JdbcTemplate is for the ConnectionCallback pattern:
It encapsulates the acquisition, use and release of a Connection. There
are however plenty of other patterns, some of which are used in the patch.
A future patch may update more of the SRM code to make use of the higher
level patterns.

The patch also replaces most uses of SQLException. SQLExcepton is a catch
all JDBC exception. It has the problem that one has to inspect a vendor
specific error code to get more detailed information about the failure.
More over, a lot of SRM code is not JDBC specific, yet used the JDBC
exception. Eg JobStorage is not limited to JDBC, yet all methods threw
SQLException.

Spring provides the DataAccessException as a replacement for SQLException.
DataAccessException is a base class for an entire hierarchy of data
access related exceptions. Spring translates vendor specific SQL error
codes to an appropriate DataAccessException subclass. The translation
happens automatically when JdbcTemplate is used.

JdbcConnectionPool is now a factory of JdbcTemplate and TransactionTemplate.
It no longer provides methods for acquiring and releasing connections.

A consequence of the patch is that many errors that were previously caught
and ignored are now propagated.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Karsten Schwank <karsten.schwank@desy.de>
Patch: http://rb.dcache.org/r/5954/
namespace: remove dead FileAttribute to NFS atts method

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
namespace: introduce CHANGE_TIME attrribute

Acked-by: Paul Millar
Target: master
Require-book: yes
Require-notes: yes
dcap: remove misterious sleep

Acked-by: Paul Millar
Target: trunk
Require-book: no
Require-notes: no
pool: fix startup for pools that saved configuration earlier than 2.2

A recent patch introduced strict checking of command options: if a
command supplies an option that is not recognised then the command will
fail.  This is limited to annotated commands.

Pools save their configuration in a 'setup' file as a series of
commands that are to be run on startup.  Unfortunately, this includes a
command with an option that isn't currently accepted. The result is
that pools with 'setup' files created earlier than 1.9.13 do not start.

This patch fixes the problem by updating the command to accept the
option but ignore it.  This is functionally the same as before
introducing the strict checking.

This patch also updates the pool defaults.  This is a set of commands
that are always executed before 'setup' to provide a base-line for
pools.  The current defaults includes the problematic command.

FOR RELEASE NOTES:

A change included in 2.6.8 prevented pools from starting up where the
pool has a 'setup' file created with a version of dCache before
v1.9.13.  This patch fixes this issue.

Target: master
Request: 2.7
Request: 2.6
Ticket: http://rt.dcache.org/Ticket/Display.html?id=8006
Requires-notes: yes
Requires-book: no
pool: Fix help text and multi-value support of PNFSID filter

Ticket: http://rt.dcache.org/Ticket/Display.html?id=7994
Require-book: no
Require-notes: yes
Target: trunk
Request: 2.7
Request: 2.6
Acked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Patch: http://rb.dcache.org/r/5998/
rpm,deb: add default limits configuration

provide reasonable defaults to run dcache.
This as root of many tickets.

Acked-by: Paul Millar
Acked-by: Karsten Schwank
Ticket: #8007
Target: master, 2.7, 2.6
Require-book: no
Require-notes: yes
srm: Resolve regression in PostgreSQL compatibility

PostgreSQL doesn't support updatable RowSets unless the table
got a primary key. Thus RequestPropertyStorage is broken as
it relies on an updatable ResultSet for a table without a primary
key.

This patch restores the old logic relying on two statements. The
entire operation is wrapped in a TransactionTemplate to ensure
atomicity of the update.

Target: trunk
Require-notes: no
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Patch: http://rb.dcache.org/r/6006/
rpm,deb: limit max open files to 65535

Good enough

Acked-by: Paul Millar
Target: master
Require-book: no
Require-notes: no
dcache (billing): rewrite triggers to eliminate multiple repeated updates

This is the first in a series of patches which seek to improve the billing insert performance.

The issue was brought forward by U. Michigan (Tier 2) when their 2.2 installation began to experience OOM in the billing domain.  After repeated attempts at diagnosis, with provisional patches deployed and observed using jConsole and their logs rsync'd to a web page, it was finally discovered that the root cause of their problem was the fact that the triggers attached to the main (fine-grained) billing tables, which derive aggregate data to be inserted into a subset of 24-hour tables, were being defeated by the way PostgreSQL implements M.V.C.C.  In particular, each UPDATE is actually a creation of a new row (rather than a change to the existing one). The old row still exists and has an older transaction id associated with it. If these old rows are not cleaned by autovacuum, then eventually we run out of transaction ids and a full vacuum is required, locking up a table completely.  HOT (Heap Only Tuples), which allows in-place updates, does so only for rows that do not contain unindexed columns.

In order to avoid multiple frequent updates, the triggers have been rewritten to check for the maximum dated entry and only insert a new row if the current time has exceeded 24 hours since that time.  The single operation is slightly more expensive, because it aggregates from the maximum time to the present, adding the appropriate missing rows, but our measurements for aggregation over 50M rows+ was approximately 1 minute, which we observed under continuous load was sustainable. In the case of a first entry into an empty aggregate table, the backward aggregation is limited to a 24-hour period.

Two notes are in order.  First, there are no doorinfo triggers, and it is only there that the datestamp frequently reflects the beginning of an operation greater than 24 hours prior to its being registered (since the request could be cached).  The storage and billing data does not usually reflect such long periods (we queried our production systems and found very few instances where the actual mover timestamp suggested a transfer lasting greater than 24 hours, and all of these were from more than two or three years in the past).  The reason that this matters is that once the new tuple/entry is aggregated, it never gets updated after that.  Given the very low frequency of these occurrences, the statistical error of the aggregated value is negligible.  If it becomes desirable to catch these cases, there are a couple of potential remedies:  increase the "window", so that we aggregate only after a longer period (say 48 vs. the current 24 hours), or we simply do an update operation on these outliers (which is slightly more costly), perhaps using a separate trigger.

Second, the triggers are not DRY.  The problem, however, is that to achieve reuse for anything more than the simple alternative values of a parameter on a given table, the function would need to be parameterized against tables and column names, requiring dynamic SQL which would then be executed, defeating the purpose -- which is to make the trigger as efficient as possible.

Finally, we have directly altered the hsqldb liquibase changesets, since the database is recreated when system-tests is rebuilt.

It should also be noted that one field on the billinginfo_tm_daily table has been altered (name changed from totaltime to average) in order to store the average rather than computing it in memory.  The changeset provides the update logic, which should not be extremely costly as the number of rows in the table will be no more than 366 per year of data taken.

A separate patch for version 2.2 will be posted.

Testing:

Built system test and ran. Deployed rpm, ran stress tests (plain SQL and full billing message passing), observing performance and queues.

Further discussion at

https://srm.fnal.gov/twiki/bin/view/DcacheCorner/BillingDatabase#PERFORMANCE_OBSERVATIONS_AND_ISS;

see in particular the final performance graphs:

https://srm.fnal.gov/twiki/bin/view/DcacheCorner/BillingDatabase#Modified_triggers

UMichigan Atlas T2 deployed the triggers on Septeber 5th and have not experienced an OOM so far:

http://head01.aglt2.org:2288/

Target: master
Request: 2.7
Request: 2.6
Patch: http://rb.dcache.org/r/5983/
Bug: http://rt.dcache.org/Ticket/Display.html?id=7915
Require-book: no
Require-notes: yes
Acked-by: Dmitry

RELEASE NOTES:

Fixes a problem with the billing database triggers which under continuous heavy loads (> 300 Hz of billing messages) led eventually to an Out-Of-Memory error in the billing domain.
webadmin: eliminate system-specific dir for unpacking of war file

The directory where the webapp war should be unpacked is currently set to /var/tmp.  This location, however, is susceptible on some systems to periodic deletion, in which case the web application fails to be found by the server.

For this version, we have simply switched this location to /var/lib/dcache/httpd.

Separate patch for 2.6 will be posted.

Testing:

Built rpm and ran instance, checking for presence of webapp directory in /var/lib/dcache/httpd, and for sanity of webadmin pages.

Target: master
Request: 2.7
Patch: http://rb.dcache.org/r/6002
Require-book: no
Require-notes: yes
Bug: http://rt.dcache.org/Ticket/Display.html?id=7979
Acked-by: Paul

RELEASE NOTES:

Fixes bug where on some systems automatic cleanup of temporary directories was eliminating the webapp application files and thus causing the webadmin pages to fail to be found.
srmclient: Fix failure detection

Addresses an issue in failure reporting for failed SRMv1 requests.

SRM v1 on master is currently broken. Our srmclient however provides a secondary
and non-informative failure:

srmcp -1 file:////bin/bash srm://localhost:8443/disk/test-1379612127-1/a
Thu Sep 19 19:35:50 CEST 2013: incorrect number of RequestFileStatusesin RequestStatus expected 1 received 0
Thu Sep 19 19:35:50 CEST 2013: java.io.IOException: incorrect number of RequestFileStatusesin RequestStatus expected 1 received 0
srm copy of at least one file failed or not completed

This is caused by the SRM client checking the number of file-level status
entries before it checks the status of the response itself. This patch resolves
the issue. The client now produces:

srmcp -1 file:////bin/bash srm://localhost:8443/disk/test-1379612984-1/a
Thu Sep 19 19:50:04 CEST 2013: rs.state = Failed rs.error = getRequestStatus(): request #-2147483645 owned by [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write / does not belong to user [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write /
Thu Sep 19 19:50:04 CEST 2013: java.io.IOException: rs.state = Failed rs.error = getRequestStatus(): request #-2147483645 owned by [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write / does not belong to user [GidPrincipal[1000,primary], UidPrincipal[1000], S:/127.0.0.1, UserNamePrincipal[behrmann], /O=Grid/O=NorduGrid/OU=nordu.net/CN=Gerd Behrmann] read-write /
srm copy of at least one file failed or not completed

I am aware of the double logging (caused by logging and throwing an
exception), however that's a different problem and not addressed by
this patch.

Target: trunk
Require-notes: yes
Require-book: no
Acked-by: Paul Millar <paul.millar@desy.de>
Acked-by: Dmitry Litvintsev <litvinse@fnal.gov>
Patch: http://rb.dcache.org/r/6019/