First pass to increase classad integers to 64 bits. #2472

Classad integers are now internally reprsented as type long long. New
methods have been added to get and set integer values as long and long
long. The soversion is bumped, but code that uses classads should only
need a recompile to use the new version.

Some work remains:
* Decide what to do when an integer larger than 32 bits needs to be
returned via the old interfaces? The options are truncate, cap, and
fail.
* Add units tests for 64-bit integers.
* Fix the random() classad function to handle 64-bit integers.
* Fix the list subscript operation to handle 64-bit integers.
* Improve the right-sift operation to not count to obscene values
and handle negative values.
Remove XML diagnosis code used only by DAGman

===GT=== #2807
Introduce a helper class to split UserLog lists

===GT=== #2807
Let submit use recursive macros

===GT=== #2807

This has been suggested a lot.

This is  a partial solution to #406 and #2428
Make schedd understand new UserLog format

This is compatible with the old Userlog format, unless ';' is used as a
separator.  There is probably some system that I don't know about uses
semicolons as a regular part of the filename

The idea here is that if the UserLog is specified in the JobAd as

UserLog = "bob;bill"

then the schedd will write a two copies of each event: one into the
"bob" file and one into the "bill" file.

I think if the event log is enabled, we get 2 copies of each event in
the event log; this is a bad thing, but I have not tested it yet.

===GT=== #2807
Make shadow understand new UserLog format

This is compatible with the old Userlog format, unless ';' is used as a
separator.  There is probably some system that I don't know about uses
semicolons as a regular part of the filename

The idea here is that if the UserLog is specified in the JobAd as

UserLog = "bob;bill"

then the shadow will write a two copies of each event: one into the
"bob" file and one into the "bill" file.

I think if the event log is enabled, we get a copy of each event for
each userlog entry. This is a bad thing; I have not tested it yet.

===GT=== #2807
Use new macro capability in DAGman

DAGman now requests the schedd and shadow to write each event to the
default log, i.e., <DAG file>.nodes.log, along with whatever log is
specified by the user.

DAGman only looks in the default log for events.

DAGman only writes POST Script TERMINATE events to its default log, not
to the user log.

DAGman specifies that the default user log is in plain text format, not
XML format.

This allows the user to have his own templated UserLog file, as
requested in #2428.

This does not completely implement the solution specified in #2807, as
we still write to a log which is also being written to by the schedd and
shadow.  However, it does fix the fd-throttling issue in #659, as there
is only one userlog that condor_dagman watches for events.

===GT=== #2807
Fix tests affected by DAGman userlog changes

===GT=== #2807
Document DAGman userlog changes in manual

And in the version history.

===GT=== #2807
===VersionHistory===
Merge remote-tracking branch 'origin/master' into gittrac-2807
Restore DAGman code for backward compatibility

===GT=== #2807
Edit version history and manual

Documented the addition of the DAGMAN_ALWAYS_USE_NODE_LOG configuration
variable.

===GT=== #2807
===VersionHistory===
Test for DAGMAN_ALWAYS_USE_NODE_LOG = False

===GT=== #2807
Filter check on macros in log file

This fixes the following use case.

The submit file is

executable = /bin/echo
mydir = bob
initialdir = $(mydir)
arguments = $(cluster)
output = 0.$(cluster).out
log = 0.$(cluster).log
error = 0.$(cluster).err
universe = vanilla
should_transfer_files = yes
when_to_transfer_output = on_exit
queue

and the DAGfile is

JOB 1 0.cmd
SCRIPT PRE 1 0.pre.sh
SCRIPT POST 1 0.post.sh

Without this patch, DAGman runs and ignores the user log specification.
With this patch, DAGman writes the specified user log file. Thus, this
also resolves #831. This returns backward compatibility by writing a
single log file instead of list, in the case above when
DAGMAN_ALWAYS_USE_NODE_LOG is false.

===GT=== #2807
Make splitting more idiomatic and secure

It was a terrible idea to tie the internal state of this helper class to
that of a variable with external state. Also, write fewer lines of code
by using the standard library.

===GT=== #2807
Accept new NorduGrid ARC job ids in the nordugrid_gahp. #3062

In the latest version of NorduGrid ARC (12.05), job ids can contain
letters as well as digits. Update the gahp's scanning of job ids to
match.
===VersionHistory:Complete===
Ticket #2541 - Squashed feature commit for classad caching.
Fix ups based on feedback #2807

This implements suggestions based on design documentation review

Remove capability to have a vector of arbitrary length fed to the
schedd; that is a recipe for a DoS. Add the "dagman_log" field; old
versions of the schedd will ignore it and thus it will only affect the
dagman user. Also, UserLog will be the same as it has always been---a
filename.

Add "DAGManNodesLog" as an attribute name.

Document this in the manual.
minor edit to 7.9.0 version history item
===GT=== #1780
Fix <SUBSYS>_DEBUG_WAIT; make debug_wait volatile. #3064