Squashed commit of the following:

commit ef02b900deb7f30f7f7d3ec1fccf4fce40f1292c
Merge: 8f77a9b 2b63d06
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon May 20 21:00:01 2013 -0500

    Merge branch 'reqmgr_wmstats_combine' into reqmgr_wmstats

    Conflicts:
    	src/python/WMCore/HTTPFrontEnd/RequestManager/ReqMgrRESTModel.py
    	src/python/WMCore/Services/RequestManager/RequestManager.py
    	test/python/WMCore_t/RequestManager_t/ReqMgr_t.py

commit 2b63d06f61d029e23655308165ed2653be68c859
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu May 2 13:35:15 2013 -0500

    add wmstats related information to reqmgr_workload_cache db

commit 8f77a9bddf39dcfdf36548b1908257d61136f4b4
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon May 6 14:13:57 2013 -0500

        add wmstats related information to reqmgr_workload_cache db

commit 24c2a0d3771e11cf88c28c9545c7c53361d84371
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri May 3 11:49:41 2013 -0500

    Fix column sizes in oracle table

    Note to self: Test all patches in both Oracle and MySQL.

commit 678b8e03a6460d143b929376d9fbdc3516a41d77
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Thu May 2 16:27:54 2013 -0500

    Clean working directories when testing patching

    The patching script can get wedged somehow by a cluttered
    workspace. Make it do a 'git clean' to tidy things up

commit aa7aa7f8839312c4e77abfe12bc02619c53c95b9
Author: Eric Vaandering <ewv@fnal.gov>
Date:   Thu May 2 15:03:36 2013 -0500

    0.9.67

    Signed-off-by: Eric Vaandering <ewv@fnal.gov>

commit e6eb94022bc87eeb53a4c43b7740fec8d4b313b5
Author: Eric Vaandering <ewv@fnal.gov>
Date:   Thu May 2 14:56:15 2013 -0500

    0.9.66

    Signed-off-by: Eric Vaandering <ewv@fnal.gov>

commit 5fcdf77a544ed5e60ff3060b670d85a2b1749c32
Author: mcinquil <mcinquil@cern.ch>
Date:   Fri Apr 12 18:26:35 2013 +0200

    Fixed version of MyProxy module UI free, with unit test

    Signed-off-by: cinquo <m.cinquilli@gmail.com>

    Adding the chance for myproxy to work only with usernames

commit 8c373b53c9e97364ca4e51913e3b5d70fc4329e4
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed May 1 16:59:37 2013 -0500

    Fix issue with DBS3 upload

    Since the latests changes an additional input
    was being sent to DBS3, they validate on every input
    and therefore blocks were being rejected. Correct that.

commit a3782af822bc5875b92b75669576296cf6e3506b
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed May 1 12:33:34 2013 -0500

    Deploy DBS3Upload in standard WMAgent

    Add configuration and changes to deploy
    both DBS uploads for DBS3 and DBS2
    in the current WMAgent

commit 1536653b75438b3f012c656aa919fec89af6c90b
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Apr 30 22:56:12 2013 -0500

    Fix queries for Oracle compatibility

    They were using a MySQL only feature in the group by
    to prevent duplicates. Do the duplicate filtering
    after the database query to keep a single compatible query.

commit a44e6614e4d12634ad50a8be6d1aa97d15801737
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Apr 30 12:42:20 2013 -0500

    Implement ACDC cleanup operations in ReqMgr

    Include the ACDC database in the ReqMgr configuration
    and cleanup collections from it when a request is moved
    to announced.

commit 08423f46f71b2e07d5e1f6e18ba2b6243ceb2b92
Author: Eric Vaandering <ewv@fnal.gov>
Date:   Tue Apr 30 10:34:41 2013 -0500

    0.9.65

    Signed-off-by: Eric Vaandering <ewv@fnal.gov>

commit c0adfd104d793834b954d6e815b2271ee5239f3a
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Tue Apr 30 10:15:31 2013 -0500

    filter requst alert

commit 60143085f3e309731bd020eefd5cdea05b3132e7
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Apr 26 16:27:40 2013 -0500

    ACDC Server CouchApp

    - Add the necessary dependencies for the acdcserver
    service for cmsweb deployment.
    - Add two new views to the ACDC couchapp which will allow cleaning operations
      based on request name and document age. Also include a basic rewrite function and
      a validate function which allows editing by the same rules as the ReqMgr couchapp.
    - Add a timestamp to all fileset ACDC records.
    - Add a script to cleanup the acdcserver in a crontab.

commit 3f3146698753dfb725b575ee2a72be7ad2f6cd73
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Apr 29 16:46:11 2013 -0500

    add detail site job information for request

commit cfdca543366645a32fc376cf1d3bff4353f799dd
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Apr 26 15:40:47 2013 -0500

    Fix unit tests

    Many changes, some tests were not updated. No changes to actual code
    just correcting the tests to the new changes.

commit c46ae6cb7a886517dd5d4651e710488b6cf0d6f4
Author: Eric Vaandering <ewv@fnal.gov>
Date:   Fri Apr 26 15:08:08 2013 -0500

    0.9.64

    Signed-off-by: Eric Vaandering <ewv@fnal.gov>

commit ef353d2fa60107ee7408c8acc9d9383ca5a4afa7
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Apr 26 11:44:17 2013 -0500

    Bugfix and new feature for TaskChain

    1. Properly include the performance information in the processing/production tasks
       in TaskChain
    2. Add the IgnoredOutputModules option for TaskChain, this allows
       an output module to be dropped in the WN. Although this is inefficient
       since the requestor could just drop the output module in the configuration instead
       after dicussing with Jean-Roch, this is useful for TaskChain where they
       have a set of standard configurations used not only for the request
       where we can drop space consuming modules but also integration
       tests where all modules are required.

commit 670ee4f2fb6283de22a1a4795494421bc5dd5b54
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Fri Apr 26 11:33:17 2013 +0200

    ReqMgr Oracle/CouchDB consistency work

    Related to #4388

    Not all Oracle tables processing yet finished.

    Modifications in the scripts done for this addition:
    https://github.com/dmwm/WMCore/issues/4388#issuecomment-17045125

commit 8b6d00bceb908c24627a28294bb2cc16936dcc5b
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Apr 25 17:30:10 2013 -0500

    Fix a few holes in the RobustMergeJobs

    Patch touches many basic places and it had some errors. These
    were catched in the unit tests and it is now fixed.

commit 69584661b79e910fbb8c99a91804e1fea9c149f3
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Apr 24 14:07:28 2013 -0500

    Cascade close-out

    Implement the option to close-out/announce requests and cascade the
    status change to all resubmission children for the request.

    For this there is a new view which allows the ReqMgr service
    to construct the resubmission tree starting from a "original" request.

    Fixes #4477

commit 4eb71ef113a211f3b1786b96f40061a151f04092
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu Apr 25 10:25:05 2013 -0500

    refactor wmstats

    commit 69b2674fc96f615d16d564bd51755b34b0324f60
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Thu Mar 21 10:57:11 2013 -0500

        remove resubmission

    commit 6118abade8a756e9d9ed76e9bde906d1c4e7d074
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Mar 18 11:49:43 2013 -0500

        fix run_closed ETA

    commit b22b791611bfb1226a3a6b1df559bd1bd7ff815a
    Author: Seangchan Ryu <sryu@fnal.gov>

    commit 5ff010448618dc9014efbe294b73e42dca151433
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Mar 13 10:55:49 2013 -0500

        acdc helper work in progress

    commit ec226ffcbb97bbf04aa7d2fb6741b8a167783e06
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Mar 6 14:48:09 2013 -0600

        acdc submission helper work in progress

commit e33951f144f3012aeb1cd445c7554609be7d3a37
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Apr 24 16:41:05 2013 -0500

    Support a many-to-many CMS site to SE relation

    Make sure that data is read properly from siteDB when building
    lists of sites from a SE, taking into account that a se can be
    shared by many CMS sites.

    Update the test emulator such that the unit test works when
    disabling the emulator in it.

commit c4cad14f6f3daa356d4dc558598d41b52cdcac2b
Author: Hassen Riahi <Hassen.Riahi@cern.ch>
Date:   Wed Apr 24 12:55:14 2013 +0200

    Fix ASO installation scripts.

commit 6d37ee0cffa11c3bae78ad9a410a85387c681075
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Tue Apr 23 16:25:01 2013 -0500

    0.9.63

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 20bd321356e2e2bf689ae3ee9697157b368b0a00
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Tue Apr 23 21:13:59 2013 +0200

    Removed GroupUser couchapp from ReqMgr unittests

    ReqMgr unittests pushing GroupUser couchapp was not necessary, removed. The
    couchapp is not used in ReqMgr at all.

commit 948a71cd1306c015b32bf6536ccaa1515cc35b0b
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Apr 22 14:17:22 2013 -0500

    Extend LHEStepZero support

    Include a new option that indicates if we are processing
    lhe input files instead of forking a generator in cmsRun,
    this way the input source options are set correctly
    to read n events per job.

    Fixes #2090

commit 9bb64799197aad6751778b47ef68e65d25b1deac
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Tue Apr 23 10:20:11 2013 -0500

    0.9.62

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 75793f3d987cba6002840cf11aaeacca07f8f570
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Tue Apr 23 16:11:35 2013 +0200

    ReqMgr - ProcessingString, ProcessingVersion

    ProcessingString, ProcessingVersion were missing in the get request response
    details. This information is stored only on spec (nor on Oracle or
    CouchDB). It may be stored in CouchDB if the request defined this at the
    injection time. Added now in the response.
    Fixes #4561

commit 90c0a3304e770fc4a9da6433918313c8351e0e88
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Tue Apr 23 14:41:05 2013 +0200

    Remove unused etc/ReqMgrConfig.py

    This configuration file is IMO not used anywhere. Only considered in the
    scenario in which ReqMgr gets deployed along with WMAgent deploy script,
    I don't think we run this anywhere. @sfoulkes, please review.

commit 980fb2fb5367f9f39c4bddbb3169f4c4f7546a34
Author: Marco Mascheroni <marco.mascheroni@cern.ch>
Date:   Mon Apr 22 18:57:54 2013 +0200

    Align Proxy to the slc6 ui

commit ba7ce742c4ebb0bc74226c3b5dd57634e697e441
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Apr 22 09:42:05 2013 -0500

    Remove site lists for second-level tasks

    Fixes #4491

commit ad5e027658ae4a42900b8dbe7ad6170a08182ebc
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Apr 18 14:15:21 2013 -0500

    Record location changes in WMBS and Couch

    Whenever a job starts running, check the CMS site
    where it started and store in WMBS and the transition
    record in the JobDump in CouchDB. This ensures that
    WMStats has the proper monitoring information and
    runtime errors are attributed to the right site.

    Fixes #3971
    Fixes #3972

commit 59e416b24b12545ae7e4ad665f75b89ec54cea9d
Author: Brian Bockelman <bbockelm@cse.unl.edu>
Date:   Wed Apr 17 14:32:15 2013 -0500

    Disregard thresholds for jobs with multiple possible running sites. For monitoring purposes, report one random site in the possible site list as the 'pending' site.

commit 2d0f23b72fdb8bdc6054f1decf2c9b1b77d47427
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Fri Apr 19 09:12:47 2013 -0500

    Revert "Fixed version of MyProxy module UI free, with unit test"

    This reverts commit 5d7ffd4be1bd5d379c11200b33edd37902e86b51.

commit f1383ef560ede3f9ee1bf1f3a33a5e007c566eec
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Thu Apr 18 15:53:00 2013 +0200

    0.9.61

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit b6558e6d0d6e1b1dfa91478283ccae13a80c8bb6
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Fri Apr 12 06:13:48 2013 +0200

    ReqMgr2 project startup

    -REST service based on WMCore.REST (new REST API)
    -some example plus request list, query REST API
    -configuration defined, called from deployment/reqmgr2/config.py
    -startup of web GUI implemented in JavaScript, calling service's REST APIs
    -couchapp ReqMgr (after initial clean up remained 1 view) shared between
    ReqMgr1 and ReqMgr2 (validation, update already previously implemented),
    necessary view will be addded
    -CouchDB database (reqmgr_workload_cache) behind, shared with ReqMgr1

    Incorporated comments from @ticonn collected on #4542
    Incorporated comments from @geneguvo:
        -only 1 major config.py under deployment
        -regexp exported into a dedicated module

    Added a check in WMCore.REST.Main, since it relies on private patched version
    of cherrypy.

commit 436be746aaca887d390dee055a0e356cdf21a5bb
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Apr 15 13:08:15 2013 -0500

    Priority handling improvement

    The purpose of this patch is to improve the priority
    handling in the ReqMgr/WorkQueue/WMAgent looking forward
    to the new global priority feature in HTCondor. It is now
    possible for priorities to cross schedd boundaries
    and be published to the negotiatior, this is
    available in the 7.9.xx series.

    The changes are the following:

    - ReqMgr
      Push priority changes to the WorkQueue by updating
      the available elements in the global queue and the spec.
      This effectively updates the priority value for any
      future split in the local queues and in any new global
      element created.

    - WMBS schema
      The priority is now added as a column in wmbs_workflow table,
      this priority is filled by the WMBSHelper the first time a workflow
      is injected into WMBS by the WorkQueue. The value used is the one
      available in the spec.

    - Local WorkQueue changes
      The scope of these changes is to prevent priority inversion due
      to the different queues available in the WorkQueue/WMAgent/Batch system
      chain. The WorkQueue now considers priority of running/pending jobs
      and queued elements before acquiring and injecting work. This means
      that if all the pending jobs are of lower priority than an incoming
      element then this will not be prevented from being acquired and injected.
      E.g. The WMAgent has 100 jobs in condor, in pending status for site A, from a workflow
      with priority 500. Also in the local queue there is 100 jobs for site A from a workflow
      with priority 500. The element in the local queue is not injected because there are already
      100 jobs of equal priority in the batch system. Now, someone creates and assigns a workflow
      to site A with priority 1000. Then that element will be acquired into the LQ
      and injected in WMBS because the 100 job slots are occupied only by lower priority jobs.

    - JobSubmitter
      The JobSubmitter now order the workflows with jobs to submit by priority and timestamp,
      this means that when slots are available the high priority workflows will be submitted
      first (after considering the task priorities). In case of equal priority, then the
      older workflow is submitted first.

    - BossAir/CondorPlugin
      The priority of the workflow is now propagated to condor through BossAir, the
      value passed to condor is a linear combination of the sub-task type priority
      and the request priority. This ensures that Merge/LogCollect/Cleanup/Harvest/Skim
      jobs are submitted faster regardless of priority to avoid missing data
      due to temporary files.
      Also the sub task path and the sub task type (e.g. Processing) is passed
      to the condor JDL as this allows more efficients updates later.

    - Resource control
      Make sub-task type priorities equal for all sites,
      the granularity creates more inconveniences down the road
      and currently there is no use case for them.
      Simplifies the priority system in submission.

    - JobUpdater (New component)

      The JobUpdater is a new component which makes sure that
      the information in the jobs already submitted to the batch
      system are up to date. At the moment this only
      concern the value of priority through the CondorPlugin.

      Additionally in the case of priority, it also ensures that
      the value is in synch with WMBS and the available elements
      in the Local Queue.

    Fixes #4509.

    Authors:

    Diego Ballesteros (@dballesteros7)
    Brian Bockelman (@bbockelm)

commit 2785cf1f098e4837e23a8e8738bc676b72de2647
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Wed Apr 17 12:24:24 2013 -0500

    0.9.60

    Signed-off-by: Andrew Melo <andrew.melo@gmail.com>

commit 6bd4a5e6f45dd86470a05cefcf98e0802bc823b2
Author: Hassen Riahi <Hassen.Riahi@cern.ch>
Date:   Wed Apr 17 11:35:58 2013 +0200

    Add ASO databases.

commit 7ebf5aa1de2c914372f430a51c44fe9ff41698c4
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Apr 16 00:57:51 2013 -0500

    Fix cases where performance parameters are None

    TaskChain requests are bugged in current cmsweb, not
    propagating valid values for the performance parameters.

    Make sure it doesn't crash the JobCreator.

commit 6b9deb7d57505bc9930b2179b746621407d3d14d
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Mon Apr 15 13:02:41 2013 -0500

    Move Jenkins' git handlers into git

    To save time when checking out, jenkins is set up to reuse a shared
    object directory with common git checkouts. This script maintains
    those checkouts. Also, change the scripts to be +x

commit d3ec0ea33c9da41449a9f6ae6e551a791c25885c
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Mon Apr 15 11:15:27 2013 -0500

    Add Jenkins bits to git

    Currently, we have a ton of scripts in Jekins input manually, this
    is a bit of a hassle since any changes need to be synced a few times.
    I'm going to begin to strip the commonly used functions into shell
    scripts stored in WMCore's git repository, so updating there will change
    everything else simultaneously

commit 3aa5ba1426007bf5ab1181d2f8c2cfd8d1aea52d
Author: mcinquil <mcinquil@cern.ch>
Date:   Fri Apr 12 18:26:35 2013 +0200

    Fixed version of MyProxy module UI free, with unit test

    Signed-off-by: cinquo <m.cinquilli@gmail.com>

commit 7ed647d4b0f845a98e2872baf1e8af6c5a6e5886
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Fri Apr 12 15:55:52 2013 -0500

    0.9.59

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 0179eaf9ccae2cd125b380374ddbb085b05a007e
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Apr 12 15:44:37 2013 -0500

    Fix broken DBS3Buffer creation in Oracle

    It was broken in 0.9.58, just a typo.

commit 5c942805fd78ece163b3a0e55c96e2d5ab49bd6f
Author: Brian Bockelman <bbockelm@cse.unl.edu>
Date:   Wed Apr 10 17:13:49 2013 -0500

    Fixed typo in test decorator. See pull #4539

commit 9659e766bb77b81bb996df91769d09b39c145dd6
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Apr 9 11:00:26 2013 -0500

    Make robust merge jobs

    Add the skipBadFiles flag to the input source in merge jobs
    in order to avoid failing the merge for a single missing file.
    When a successful merge job comes back with skipped files
    these are added to the ACDC collection for the merge task
    for posterior recovery.

    Additionally, now CMSSW steps will be declared failed
    if no output files are produced. This way the scenario
    where all files were skipped is not counted as a success and also
    allows to catch glitches when the cmsRun process fails to process the input
    but stil returns with exit code 0.

    This is only enabled for ParentlessMergeBySize.

    Fixes #4476

commit 84b67aa463eb145d7da39df13a8fea34091c7227
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Apr 9 15:38:59 2013 -0500

    Store correctly the SizePerEvent and friends

    Make sure that SizePerEvent is stored correctly in CouchDB and Oracle
    if present, it goes in hand with RequestSizeFiles and RequestNumEvents
    but these two are not really present in any request schema
    so they will be always zero, only SizePerEvent is mandatory.

    Also fix the clone and assign of requests pre-upgrade.

commit 2ff0696bcdf42002a645afc90164572fb40c2a3e
Author: Brian Bockelman <bbockelm@cse.unl.edu>
Date:   Mon Apr 8 10:11:36 2013 -0500

    Implement a reasonable API to get the details - events, lumis, parents - of each file of a dataset

    Conflicts:

    	test/python/WMCore_t/Services_t/DBS_t/DBSReader_t.py

commit d02a616c4a5740f3601a6f4ccd0ef9686cca9764
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date:   Thu Mar 7 06:03:21 2013 +0100

    changes to DQM harvesting

    Change the Harvest job splitter to have two distinct modes, Periodic and EndOfRun that do not overlap.
    In StdBase automatically add an additional EndOfRun DQM Harvest task if a Periodic DQM Harvest task is
    specified. The EndOfRun task will trigger when the Periodic task is complete.

commit 17e897ffcb384d33f93a4ceb29efaec044a0d724
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Fri Apr 5 14:06:13 2013 -0500

    0.9.58

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 173f8016382fccf017c9e08879ea690670e190f7
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Apr 4 16:41:35 2013 -0500

    Catch empty lumi runs

    Improve on 851acb4d94fc20d07d15b6aa3fe94d82c43aa2ed to catch
    the cases where the runs dictionary is not empty but the lumi lists
    are empty for a run in the report.

commit 353a3f0d7f68d82fefce8bf041753327b31290c7
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Mar 27 09:55:45 2013 -0500

    Include performance estimates in Condor JDLs

    On the way to better provisioning, some of the splitters are able
    to "predict" how long a job will take based on time per event estimates,
    how much disk it will require based on estimates of output size per event
    and how much memory the job will require. This information is included in
    the condor JDLs in the following values:

    +MaxWallTimeMins : Time in minutes, used already by CRAB2, coordinating with the FE admins if central production is already set to use it.
    request_memory : Standard condor attribute to request memory during the matchmaking. It's in MB
    request_disk : Standard condor attribute to request disk space during the matchmaking. It's in KB

    The list of algorithms which support estimates follows:

    EventAwareLumiBased, EventBased, LumiBased, FileBased: Provides estimates for all 3 values, this covers all Processing/Production/Skim tasks.
    WMBSMergeBySize, ParentlessMergeBySize: Provide very accurate estimates for the disk space. This covers all merge tasks.

    The tasks which provide no estimates are:

    Harvest, Cleanup, LogCollect : All these tasks are very fast and with very little requirements.
    We may tailor some defaults for them in the future.

    The initial idea included changing the estimates for time per event, size per event
    and memory dynamically using couchDB information. However after giving some thought
    I realized that with the current schema this is not useful and can be harmful. The case
    is when the second block is split after only fast jobs have run, these will have
    low time per event and memory usage which will trigger bad pilots for future jobs
    and won't allow for correction.

    The estimates will be only reliable in a model where we first fully process at least a block
    before splitting the rest of the workload. This would give some guarantee that the sample
    used for the performance information is somewhat representative.

    Fixes #4472

commit 91f836c73724c139ce6f578c7ffb626bb8448b28
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Apr 4 15:57:07 2013 -0500

    Avoid re-running performance killed jobs

    Improve the exit code handling in the ErrorHandler,
    retrieve a list instead of a single random non-zero exit code.
    Compare the list against the exit and pass list.
    Change the exit codes for performance kills to make them identifiable,
    include them in the default configurable list or exit codes.

    Fixes #4473

commit bf01cfb3674e7dc0c53b44f16ca43d95faeae443
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Thu Apr 4 21:08:19 2013 +0200

    0.9.57

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit e231b1ecccbd2c3d86469e96de47ebe4c728294e
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Apr 4 13:24:18 2013 -0500

    Fix DbsUrl for clones

    Make sure the correct DbsUrl is stored in couch, even if not present in
    the schema. However, if a non-null value is in the schema store that one.

    The unit tests in WMCore_t.RequestManager_t.ReqMgrWorkload_t test the following cases:

    - If no DbsUrl is present in the schema, then the default from the specs is stored in couchDB.
    - If DbsUrl is present in the schema, then it is stored in couch regardless of the request type.
    - If no DbsUrl is present in the schema, and the workload doesn't provide a default then couchDB stores a null value.

    Fixes #4498

commit d8750bf748c4d38da0d10a7209e33ef849729883
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Thu Apr 4 16:46:54 2013 +0200

    ReqMgr - removing wrong DBS URL from configuration

    ReqMgr configuration defined:
    dbs3 = 'http://vocms09.cern.ch:8989/dbs'
    which is wrong. Was removed along with related code in the Announce module.
    The results of handleAnnounce call here are at least questionable, pending Ops
    feedback it may be removed altogether.

commit 744557409952b23ac0c8192789f585dd2585fd22
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Apr 3 17:25:14 2013 -0500

    Fix SiteDB service for resource control

    It was broken for the call to include all T1s/T2s after
    the  changes for SiteDB v2.

commit f387f6f747f260193928d880abf0aca9a3ac9de8
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Wed Apr 3 13:36:47 2013 -0500

    0.9.56

    Signed-off-by: Andrew Melo <andrew.melo@gmail.com>

commit f71433110fa203919238780b9fa6258e0e0123e6
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Apr 3 12:08:16 2013 -0500

    fix running close eta

commit b35b6432e8bbbb424de4d7950bf928274d80d685
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Apr 3 13:11:37 2013 +0200

    ReqMgr fixed unittets for #4516

    Forgotten to re-run the unittests, fixed.

commit 57d6ebe73ddcdbbabb112ee293848b54de9e24a2
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Wed Apr 3 12:24:47 2013 +0200

    0.9.55

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit d1112c957063e49fdf2f07f90f861506342c9aee
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Apr 3 12:15:04 2013 +0200

    ReqMgr request parameter DbsUrl stored in CouchDB

    Correction of #4515
    e.g. for MonteCarlo workload the DbsUrl was not really stored in CouchDB, even
    though it was specified at injection, querying the request was returning None
    for DbsUrl. As discussed with @efajardo, DbsUrl argument, once specified,
    shall be stored for all types of workflows.

    Fixes #4498
    Correction of #4380 (DbsUrl should not have been removed, only fixed so that
    it's properly stored in CouchDB, as done here now)

commit 2fcb4aa3f1b674ff790f5a982e2b21f699dfc779
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Wed Apr 3 10:11:47 2013 +0200

    0.9.54

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit e1dff41218acf3be0da4708f59a0dfe7c7a55a5a
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Apr 2 16:04:23 2013 -0500

    Restore DBS Url in CouchDB

    Also fix the issues with the DBSUrl and the clone API

    Fixes #4498

commit 67590f3cb01980243e24c6c0144e8dffd46095cd
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Tue Apr 2 12:14:24 2013 +0200

    0.9.53

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit fdc232dffe64c979c348d5e1e1868e7e78f07f95
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Mon Apr 1 17:13:20 2013 -0500

    0.9.52

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 398eff6e0192ad13c32b03f62fe725edbb7b81e2
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Apr 1 16:21:14 2013 -0500

    Squashed commit of the following:

    commit e1f1265bbdb45b23e88900808803da789f17c27a
    Merge: 44aedc1 146d10f
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Apr 1 15:47:37 2013 -0500

        Merge branch 'acdc_support' into pull_april

    commit 146d10fd49de01a170986dd4fbb299eeef0f5bbb
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Apr 1 15:45:49 2013 -0500

        add acdc_support

    commit f1f4a0c08013de1fec5c6d83e920239283cbfd3e
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Thu Mar 28 12:00:41 2013 -0500

        fix runnning-closed ETA

    commit 463029de145708648806060666fcc7093116082d
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Mar 27 13:25:30 2013 -0500

        add control fix

    commit 69b2674fc96f615d16d564bd51755b34b0324f60
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Thu Mar 21 10:57:11 2013 -0500

        remove resubmission

    commit 6118abade8a756e9d9ed76e9bde906d1c4e7d074
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Mar 18 11:49:43 2013 -0500

        fix run_closed ETA

    commit b22b791611bfb1226a3a6b1df559bd1bd7ff815a
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Mar 13 16:31:53 2013 -0500

        acdc

    commit 5ff010448618dc9014efbe294b73e42dca151433
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Mar 13 10:55:49 2013 -0500

        acdc helper work in progress

    commit ec226ffcbb97bbf04aa7d2fb6741b8a167783e06
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Mar 6 14:48:09 2013 -0600

        acdc submission helper work in progress

commit 4f9a1064609f19572f0e7e4133355f8f4ce519f5
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Apr 1 15:55:53 2013 -0500

    add requestor if not specified

commit 838c32e2ad1d0e4228e1bdfe51d9a310ddf11913
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Apr 1 11:22:52 2013 -0500

    Fix the initial location for pileup

    It was set to the SEs instead of the CMS site names. Correct that.

commit 7d89b8cba64d54673106cb36d13a1e78a3417eb1
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Mar 22 13:57:09 2013 -0500

    Include performance parameter in workload

    First part of #4472, include the performance information that is now already
    required but only stored in the schema. The performance information is stored by
    task and can be different depending on the type of task.

    Tasks like skims and second step TaskChain use their own estimate which is optional
    and defaults to the top level value.

    This is only the ReqMgr and Workload definition changes. After this we need the changes at the
    agent level in the splitters but that is independent of cmsweb schedule.

commit 6795078f3b555225febef1ae79f45e7f432991d9
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Apr 1 10:42:25 2013 -0500

    Add additional dependency for workqueue

    Needed for tracking the pileup datasets

commit 650a928737518ab226763b39bb44d8da38b82da4
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Mar 11 01:44:18 2013 -0500

    Block close settings per request

    Move the settings that determine when to close a produced block in
    DBS to the request level instead of an agent-wide parameter.

    These settings are:

    - BlockCloseMaxWaitTime: Maximum time that a block will be left open since its creation.
    - BlockCloseMaxFiles: Maximum number of files in a block, when reached the block is closed.
    - BlockCloseMaxEvents: Maximum number of events in a block, when reached the block is closed.
    - BlockCloseMaxSize: Maximum size of a block, when reached the block is closed.

    Breakdown of the changes:

    - At the RequestManager and workload definition level:
      - The WMWorkload contains the 4 block closing settings in the properties section (top level).
      - These are set to sensible defaults that are also modified per request type. E.g. LHEStepZero workloads have 5 files per block only.
      - The values can be modified through the assign page, these parameters are optional. The identifiers of the parameters were given above.

    - At the local workqueue level:

      - The WMBSHelper now installs all tasks that will potentially produce merge output in the dbsbuffer workflow table, this includes the block closing settings.

    - DBSBuffer3 schema changes:

      - The dbsbuffer_workflow table now contains 4 columns, one for each setting.
      - The InsertWorkflow DAO is updated to require the settings with any workflow injection
      - A new UpdateSpec DAO is provided to updated the spec path in the dbsbuffer_workflow table

    - JobAccountant changes:

      - Since the workflows are injected at the WorkQueue level then workflows are not injected by default when files arrive,
        however if no workflow is found, the JobAccountant can still take care of this (so far the only use for this is unit testing).

    - DBSUpload and DBSBuffer changes (2 & 3)

      - Every file record used in the DBSUpload pollers now contains the block closing settings
      - New blocks take the block closing settings from the first file they add and then use it to check when the block should be closed
      - Any component setting has been removed

    - Unit tests
      - New emulator for the DBSInterface module for DBS2, this can be used to test the full dbsUploader without having a valid DBS instance
      - New mock API for the DBS3 Api, this can be used to the test the full dbsUploader (DBS3) without having a valid DBS instance
      - Extend unit tests for the dbsUploader to test new settings per workflow in DBS2 and DBS3

commit 346de1314fbeea2e8eb5a5940e691e12d36fdbc9
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Mar 29 11:20:46 2013 -0500

    Track pileup locations

    The WorkQueue now tracks the location of pileup datasets
    in a workload. The handling of these datasets it's similar
    to the parent data, it is required that all pileup datasets
    are at a site before acquiring the work for that site. However,
    since the pileup is set at a dataset-level instead of blocks.
    It is required that just a block is at the site for the dataset
    to be considered "at the site". If the pileup is not at any
    of the valid sites (after white/black lists) then the work
    will wait in Available until the datasets/blocks are transferred.

    Fixes #3733

commit 76d631023c6e351c93616d14b6e45119c493aa02
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Mar 27 21:59:24 2013 +0100

    ReqMgr CouchApp validation function

    Fixes #4464

commit 1de17b2dcde06894f7731ced29c3197452525f6f
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 27 13:45:23 2013 -0500

    fix firefox in Mac and Windows

commit 7485e3213634d63b3f2bdbbde7315c1d02fd9063
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Mar 27 13:19:08 2013 -0500

    Fix xrootd option for ACDC

    Allow the trust site whitelist as location option to work for ACDC workflows
    the ACDC files will have locations according to the site whitelist and blacklist.

    Also improve an ACDC view making it compatible to newer versions of couchDB where
    documents can't be modified in a view (it was sorting in place a element of the doc
    inside the view).

    Fixes #4496

commit 0efc87c249906fa3d5bba913307bb497200e486a
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Mar 27 16:48:04 2013 +0100

    ReqMgr GetRequest DAO not propagating all values

    Fixes #3437

commit 848cb04d48659e0fab746c4ebc8b7b1471ab01b1
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Mar 27 14:09:50 2013 +0100

    Second round of ReqMgr CouchDB values fixes.

    RequestSizeEvents - in majority of couch requests not defined, no references
        in the code (probably was at some point), should be removed from all Couch
        requests ; deprecated, injection fails will fail now if specified
    RequestEventSize - deprecated, injection fails will fail now if specified,
        should be removed from all old Couch requests
    SizePerEvent - must exist, parameter is mandatory, from Oracle is taken from
        reqmgr_request:REQUEST_EVENT_SIZE (mess that the above probably existed
        when someone was introducing SizePerEvent)
    RequestSizeFiles - made mandatory, handled properly so that values are stored
        in Couch, on old requests values should be loaded from
        reqmgr_request:REQUEST_SIZE_FILES
    RequestNumEvents - made mandatory, handled properly so that values are stored
        in Couch, on old requests values should be loaded from
        reqmgr_request:REQUEST_NUM_EVENTS
    PrepID - made optional, properly stored in Couch, on old requests values
        should be loaded from reqmgr_request:PREP_ID

    All unittests updated. reqmgr.py now contains checks for the above.
    Database Oracle -> CouchDB migration/consistency scripts should implement the
    above when migrating values.

commit 8a34aa48a0195f080d34bff4dfc75c83ee2bd25a
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Tue Mar 26 23:44:35 2013 +0100

    First round of ReqMgr CouchDB values fixes.

    First round of ticket 4388
    https://github.com/dmwm/WMCore/issues/4388#issuecomment-15390453

    Removed from CouchDB:
    "ReqMgrGroupID"
    "ReqMgrRequestID"
    "ReqMgrRequestorID"
    "ReqMgrRequestBasePriority"
    "WorkflowSpec"

    Unittests and reqmgr.py checks that these request parametrs never appear in
    CouchDB on newly created requests.

    "RequestWorkflow" is left on requests, contains URL of the spec file, though
    it's redundant, parameter is however forbidden on injection and is set by
    ReqMgr. Spec manipulation depends on presence of this argument. Too many
    changes would be required to get rid of it. reqmgr_request:worklow would be
    ported if necessary to Couch or removed completely in ReqMgr2.

commit ebfd06162aa1e3d506d65f188de411737ca29b5f
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Mon Mar 25 16:10:14 2013 +0100

    Update to ReqMgr Oracle/CouchDB consistency tools

    -description in the README file
    -couchdb consistency checker and correction started
    -couchdb dumper
    -frequent running of Oracle, CouchDB database consistency (e.g. the same
    number of requests in both databases) checker made easiler

commit 5c863124d313e400ae007d618791995f787abf19
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Fri Mar 22 13:56:24 2013 +0100

    Fixed ReqMgr links, user/group priorities removed

    - some links to reqmgr/view broken (Template not found) - fixed, offers
      default ReqMgr Browser page
    - DAOs, APIs and all related code to user and group priorities completely
      removed, as confirmed by @efajardo, Ops have never used it (and it would
      have hardly worked if they had)
    - yuiserver link removed

    Someone (@efajardo) should comment if links to GlobalMonitor or broken
    "monitorSvc" should be kept on the ReqMgr page.

commit 4c035c39a80ce39666e64863e3d6104dadce2640
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Thu Mar 21 19:42:51 2013 +0100

    Removed User, Group, Base request priorities

    Removes handling of
        ReqMgrGroupBasePriority
        ReqMgrRequestBasePriority
        ReqMgrRequestorBasePriority

    Oracle tables values are still there and Oracle database wasn't and won't be
    altered, but above values are not used anymore in the code. Only a single
    inject and request parameter 'RequestPriority' is now used. It's value is
    updated absolutely, not by offset as it probably was the previous intention
    (as agreed via IM by @efajardo).

    Value of reqmgr_assignment:priority_modifier from database definition - also
    ignored, as well as a bunch of priority related database table columns, as
    said above, except for reqmgr_request:request_priority value. This always
    agrees with CouchDB "RequestPriority" document field.

    unittests and reqmgr.py --allTests OK

    Fixes #4465

commit ebe436b9c2b82f6b8908c9ebff297790cc29cc0e
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Mar 20 17:02:35 2013 +0100

    Fix wrong ScramArch info on view details page

    -Fix wrong ScramArch info on view details page: set of values was nicely
    hardcoded in the page definition regardless of the actual values in the
    request, shows it correctly now

    -CMSSWVersion, ScrachArch and GlobalTag were made immutable on this page.
    ScramArch could not have possibly worked anyway.

    -links from view/details page to Original Config and TweakFile would yield
    Server Error 500 if a request was created with ConfigCache document coming
    from a host different from where ReqMgr runs (using ConfigCacheUrl argument).
    This was fixed too.

    Fixes #4249

commit 80e0a1373acfdfc9657bdb005114acc88fdcf54c
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Mar 12 15:51:53 2013 -0500

    Add support for the splitting page in reqmgr.py

    Add basic support for changing the splitting
    with the reqmgr.py script, it is available
    through another section in the JSON files.

    Also incorporate splitting change in the
    allTests procedure.

commit 86c9b63894dd3a69ef31cf7d6d6ec09e768279e0
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Tue Mar 19 15:34:01 2013 +0100

    0.9.51

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit b11f5ed60c2db01910e51eb41facb70e54a8d777
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Mon Mar 18 12:09:12 2013 -0500

    fix missing input files

commit 535a46c22a3929551c0f86b4034298f73dc8c1cf
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Tue Mar 19 14:47:05 2013 +0100

    Fixes automatic testing of TaskChain requests

    TaskChain requests JSON template files had CouchURL argument defined. Which
    overrides ConfigCacheUrl. This could have possibly worked only on the defined
    host which was testbed, but failed with CouchInternalError otherwise.

commit b53e4dabbc1463b514fbe482dcd9f915ab936025
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Mar 18 16:22:28 2013 -0500

    Add TransientOutputModules to TaskChain

    Same functionality as in ReReco, some modules can be kept unmerged if
    necessary in order to reduce the output but allow subsequent processing.

    The parameter per task is TransientOutputModules, also KeepOutput is still
    valid and can be used to wipe all the output in the task to unmerged. If
    KeepOutput is False then TransientOutputModules doesn't have any effect.

    Fixes #4474

commit 1fd34b7f0d832b908cc434b1c9915c17a254c245
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Mar 14 17:33:14 2013 -0500

    Fix most unit tests

    Several unit tests were broken because of broken logic, not infrastructure
    issues or edge cases. Fixed those to have a cleaner testing environment.

    Removed most test from the allowed_failing_tests. The only survivors
    are the ReqMgr tests which I can't figure out why they are broken in Jenkins only
    and the REST tests which are broken locally too. Someone with more knowledge
    of the REST stuff can fix those. Here is the breakdown of changes an explanations:

    WMCore_t_ResourceControl_t_ResourceControl_t_ResourceControlTest_testInsertAllSE : New SiteDB emulator had a broken function.
    WMComponent_t_RetryManager_t_RetryManager_t_RetryManagerTest_testA_Create : Create cooloff transitions to created instead of new
    WMComponent_t_RetryManager_t_RetryManager_t_RetryManagerTest_testY_MultipleIterations : Create cooloff transitions to created instead of new
    WMBS_t_WorkQueueTest_testWorkQueueService : Unit test was missing the creation of the BossAir schema
    WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase.WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase.testWhiteBlackLists  : Unit test was broken after latest fixes
    WMCore_t.WorkQueue_t.Policy_t.Start_t.ResubmitBlock_t.ResubmitBlockTest.WMCore_t.WorkQueue_t.Policy_t.Start_t.ResubmitBlock_t : All tests broken after SiteDB emulator change
    WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase.WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase.testWhiteBlackLists : Tests were not properly resetting the defaults after every test.
    WMCore_t.BossAir_t.GLiteLIParser_t.LoggingInfoParserTest.WMCore_t.BossAir_t.GLiteLIParser_t : Needed test files in a big commit with no apparent reason, restored them.
    WMComponent_t.AlertGenerator_t.Pollers_t.Agent_t.AgentTest.WMComponent_t.AlertGenerator_t.Pollers_t.Agent_t.AgentTest.testComponentsPollerBasic : Can't assure no children condition in Jenkins
    UserFileCache_t.UserFileCacheTest.UserFileCache_t.UserFileCacheTest.testChecksum : Needed test files in a big commit with no apparent reason, restored them.
    WMCore_t.Services_t.WMStats_t.WMStats_t.WMStatsTest.WMCore_t.Services_t.WMStats_t.WMStats_t.WMStatsTest.testWMStatsWriter  : Change in response message broke unit test
    WMCore_t.Storage_t.Plugins_t.DCCPFNALImpl_t.py : Test module and module-under-test are broken. Fixed imports only and added test to blacklist
    WMCore_t.Storage_t.Plugins_t.SRMV2Impl_t.py : Test module and module-under-test are broken. Fixed imports only and added test to blacklist
    WMCore_t.Services_t.pycurlFileUpload_t.py : Fixed updated imports and test functionality
    WMCore_t.Services_t.DBS_t.DBSReader_t.DBSReaderTest.WMCore_t.Services_t.DBS_t.DBSReader_t.DBSReaderTest.testGetDBSSummaryInfo : Test was not marked properly as integration
    WMCore_t.WMBS_t.JobSplitting_t.ParentlessMergeBySize_t.ParentlessMergeBySizeTest.WMCore_t.WMBS_t.JobSplitting_t.ParentlessMergeBySize_t.ParentlessMergeBySizeTest.testLocationMerging : Test was outdated compared to the setup method.

commit adfb4bdfc477fb56b760a9d4aad2edf5a79e5196
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Mar 14 13:59:11 2013 -0500

    Restore the T2_XX_SiteA/B/C sites, SEs and CEs

    This is needed for WorkQueue unit tests to work

commit 68bc690abf9bca3b47bcaa5a104f7bd8440e89c5
Author: Manuel Giffels <giffels@gmail.com>
Date:   Wed Mar 13 11:14:23 2013 +0100

    Fix broken unittest caused SiteDBv2 emulator

commit a8f0c0bb74493c8c9c5585464377e944b9067d73
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 13 11:06:51 2013 -0500

    handle when couch has problem. keep accountant going and retry logcollect accounting

commit ce6588c25b9821335e5be803cbd576908cf8cc14
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Mon Mar 11 13:36:32 2013 -0500

    0.9.50

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 164f27e4b23c7f127ed3e0006b96f4b0d9148a87
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date:   Fri Mar 8 21:47:15 2013 +0100

    fix LSF submission when overloaded

    When LSF is overloaded, the LSF bsub submit command can print out some debug messages to stdout before submitting the job. In these cases the LsfPlugin does not detect that a job has been submitted. Fix this.

commit fbca67a62cf9ab0943c786b66840b5f8e49d962e
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Fri Mar 8 15:32:37 2013 +0100

    ReqMgr2 - CouchDB request docs null fields

    * some fields removed from CouchDB (useless)
    * majority of other CouchDB fields is not properly set and updated (e.g.
      Status or Priority as changed on the request)
    * reqmgr.py --allTests checks Oracle, CouchDB consistency, at least partly now

    Fixes #4380

commit 3693437e93c1a92ae06d493117f51a77509b00c4
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Mar 7 17:18:15 2013 -0600

    WorkQueue fixes

    Check that replication is alive in the WorkQueue, kill if it is
    in error state so that it can be restarted.

    Also make a temporal fix to the WorkQueue replication filter
    since we have _deleted documents with data in cmsweb which we don't
    want to shuffle around.

commit 834617921fb2ae8fe5c91dd9665057b7af326dc4
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu Mar 7 22:30:29 2013 -0600

    add the unittest

commit 8a970e0f95c9397ee9527195e018079390f8bbd5
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Mar 6 21:00:12 2013 -0600

    add acdc and agent info in job document

commit c24c1b664d8c67e81e0cbc2785377790c4d14739
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Mon Mar 4 16:08:26 2013 -0600

    0.9.49

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 893bc70fb76102007f46c90f088d34662a67d2a9
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Mar 4 10:58:53 2013 -0600

    Fix straight to merge when splitting is changed

    If splitting is changed through script/pages and there is a
    transient output module, the property of disabling
    straight to merge is broken by the change which can
    cause spurious output in merged area from straight-to-merge
    mechanisms.

commit 38d3d88dec9883d2b66cbc88b427837a3269a560
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Feb 28 15:45:41 2013 -0600

    Fix paused jobs script.

    Add confirmation on fail jobs and don't crash when no paused jobs
    are found.

    Fixes #4392

commit e50fb43a241c0fc6263e0f50574334ba57bdf2a2
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Thu Feb 28 14:59:46 2013 -0600

    0.9.48

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit fa842d86b5ffa0482a9c722c144bb5f22982f195
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Thu Feb 28 14:57:17 2013 -0600

    Fix my revious mistake, recommit the change log for 0.9.47.

commit 064eb0d2476a083f62b61b8ef730125ee7ac51ee
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Thu Feb 28 14:56:10 2013 -0600

    Revert "0.9.47"

    This reverts commit d70ccf2bbadc6720f6974d33b3a91e86e1aa945c.

commit cb6265c6bb44fe9161c5c301ffaa649bfbbea27d
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Thu Feb 28 14:15:26 2013 -0600

    0.9.47

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 3be2c6a7e0bf436962f9c4d5f51672aeaed734d6
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Thu Feb 28 13:56:35 2013 -0600

    handle acdc campaign when campaign is passed in the workload summary
    handle object for expand format

commit 1e92f0ffe1a217d70af2526588ed64d9d1e108d6
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Wed Feb 27 16:16:27 2013 -0600

    0.9.46

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit f9d90b118ea130d8529a540407e4323fda6a277c
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Tue Feb 26 14:16:12 2013 -0600

    Squashed commit of the following:

    commit 1fb0e573580a59e8f3398810027102b030349668
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Tue Feb 26 14:05:33 2013 -0600

        reformat request summary

    commit a9e58d0f97783086d93efa381c9239c7889aa32c
    Merge: f334328 9e736eb
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Tue Feb 26 11:54:37 2013 -0600

        Merge branch 'update_using_post' into vocms236_3

        Conflicts:
        	src/python/WMCore/Services/WMStats/WMStatsWriter.py

    commit f3343288a3c28d2a222faa68065fbdc2fbc4831e
    Merge: 5a45e53 da6473f
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Tue Feb 26 11:51:17 2013 -0600

        Merge branch 'logcollect_assoc' into vocms236_3

    commit da6473f483565015c18242cd75221d34943aa1db
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Thu Feb 21 15:54:18 2013 -0600

        add logarchive lfn to wmstats job summary

    commit 9e736ebb294df55c93b7b5bc867cc7222d2f80d5
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Feb 18 14:09:29 2013 -0600

        update agent info using request body to prevent uri size limit

commit 125d15cbc3921f5033b8a58544ca488b40c49018
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Feb 27 11:21:59 2013 -0600

    update campaign when acdc inherits

commit 7167f28042c64287652000751d1c7c7b69b6ac42
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Feb 26 12:56:10 2013 -0600

    Check for run/lumi info in CMSSW outputs

    All outputs from the CMSSW step should have run/lumi information
    from the cmsRun process. Verify that and if not present
    fail the job.

    Fixes #4416

commit f7d3a77da6a5eba91dd3986bd7ee582f520515ac
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 25 17:39:09 2013 -0600

    Fix prioritization of tasks

    The tasks priorities in batch level were being taken from a task
    value that was not set for all tasks and also was static to
    5 for Merge/Cleanup/Harvest and 0 to Processing.

    ResourceControl allows dynamic priorities per site and task type,
    this patches changes the submitter to use those instead and
    remove the signature of WMTask.taskPriority since it is not
    needed anymore.

    Fixes #4417

commit aa70c7ca70fb2e6f891d1682180e6de912bc1ecc
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Feb 22 02:08:21 2013 -0600

    Improve site states functionality

    The possible site state list has changed, the Finalizing state doesn't exist anymore.
    There is a new state labeled 'Aborted'.

    Normal state functionality has no changes.
    Draining state doesn't pull work for existing workflows anymore into the local queue.
    Down state funcionality has no changes

    Aborted state:

    This new state indicates that a site is in error state and won't be recovering soon,
    the best approach then is to kill any job at the site and fail any jobs that can only
    run there since it is not efficient to wait until the site comes back or the input
    for these jobs is lost at the site.

    - If a site is in Aborted state, the JobSubmitter will not use it for submission.
      Moreover if a job can only run at an Aborted site then this job will go into
      submitfailed state with errorCode 61102.
    - When a site is changed to Aborted state, all the jobs currently assigned to that site
      in the batch system will be killed, the error code for this is 61301. These jobs
      will follow usual WMAgent retry mechanics which means cooloff/resubmssion. Note
      that when resubmitted they will fail if they can't run anywhere else as stated above.
    - For WorkQueue purposes an Aborted sites behaves as a Down site.

    Additionally to support this functionality, there are changes in the ErrorHandler to improve
    the smart error handling, now there are certain not-configurable errorCodes which cause
    immediate exhaustion of the jobs. 61301 is the only one as of now and also if a report
    has no time information it will now check the exit codes before deciding the fate of the job.
    The ErrorHandler now reads FWJRs by default and has a default maxFailTime of 32h.

    The error codes are now contained in WMExceptions.py, just a few of them now. But more
    will be added later to keep them documented.

    Finally, the JobSubmitter now resets the cache when a site changes state, this is to
    ensure that the jobs are assigned correctly according to site availability at all times.
    Although it can make the JobSubmitter slower on agents with a large number
    of created jobs, however site state changes are not frequent so this is tolerable.

    Fixes #4408

commit 78d72d82eb86c9fd3b710b6e16353133bd87b515
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 25 16:40:43 2013 -0600

    Bugfix for file information

    In some occasions the TaskArchiver fails to delete some file records
    correctly from WMBS and some records are left with the default
    values in wmbs_file_details. If a second workflow comes with the same
    input files it may find zero events files in WMBS which causes
    problems in the splitting algo like EventAwareLumiBased. Fix
    this by updating the file information in WMBS when trying to add a new file
    that is already there. This rarely happens so there is no real cost.

commit 92da09e210280646303ad39cdeebde48d1c4de54
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Feb 27 10:19:10 2013 -0600

    Fix CMSSW diagnostics of segfaults

    When a SEGFAULT occurs in cmsRun, the process exits with a 134
    error code and leaves an empty FrameworkJobReport behind,
    the same case as when the cmsRun process receives a SIGKILL or SIGTERM.
    The empty FWJR caused the diagnostics handler to crash and therefore
    the job didn't do the logArchive step or returned the report pickle,
    this patch fixes that and extends the unti tests for the CMSSW executor
    with diagnostics.

    Fixes #4134

commit a184b979858ba9559a693bcdf363c351cec87ea6
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 25 15:42:12 2013 -0600

    Fix ACDC for production

    Improve the EventBased splitting algorithm to support ACDC for production
    workflows (MonteCarlo and LHEStepZero only). It implements a limited
    ACDC where the jobs will be reproduced exactly as the last time. For recovery
    of statistics it is suggested to use the statistics extension feature instead.

    Fixes #4399

commit 5d5862f08344f2dd9501b540db33150170cabf1c
Author: Diego <geneguvo@gmail.com>
Date:   Tue Feb 26 14:52:32 2013 +0100

    Remove the check for the local wmc-dist-patch script as it is not shipped with the project source code. It gets included only during RPM build time.

commit 3e235fefa59f614f9bc914aa76ed6f4f8be0c7d7
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Jan 24 17:36:16 2013 -0600

    Include task type priority in LSF jobs

    Specify job priority according to task type priority, this allows
    LSF to do a better scheduling of the jobs.

commit 1498f6bba1d1c04f4d0a0a3b4492f31a17cfc0cf
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Mon Feb 25 09:15:50 2013 -0600

    0.9.45

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit b8251d2ef2c7a81c0f86dc3245eb7ae7e638d639
Author: Samir Cury <samir@cern.ch>
Date:   Mon Feb 25 11:39:46 2013 +0100

    Adds unit test for dashboard reco performance reporting

commit 51711f3e5d22c7910fbd609c0bf3a8fb22b481e7
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Feb 22 17:02:32 2013 -0600

    Bugfix for Global Workqueue

    When splitting a large dataset (>100 blocks) the workqueue can blow up and leave
    the requests in funny states. This comes from the latest additions triggering
    the max size of an URI in an update call to couch.

commit 428e6cd09319b5ac828ffe7cda58024c34eff510
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Wed Feb 20 15:28:59 2013 -0600

    0.9.44

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit f9ec3d60fae86e0e5d049bb9ff40b9ae4c7f4e9d
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Wed Feb 20 14:02:11 2013 -0600

    Bugfix in WMStats update document

    The last status for aborted requests is aborted-archived. Fix a typo in the update function.

commit 705b8de3b7ce9524d120638e3baea4939b289d31
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Feb 20 13:19:21 2013 -0600

    Squashed commit of the following:

    commit b7a926efa7666a38ca6c5bcdd5d820ca189dc4e2
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Feb 20 13:17:02 2013 -0600

        GUI change

    commit a3c13ec92c911b214c4132ee53b13295747852a7
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Feb 18 10:53:56 2013 -0600

        march deploy work in progress

    commit 6c7cfa0f08e02e5c50bcc401a8d5d017b3457306
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Tue Feb 12 10:27:27 2013 -0600

        change code to datatable library 1.9.4 and jquery and jquery ui version to upto date

    commit bca64eda8395b565a28be09e8aecfb006d1b07fa
    Merge: 3777e28 df683cf
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Feb 11 15:25:34 2013 -0600

        Merge branch 'agent_info_detail' into 2013-2-11

    commit 3777e2856f21e35f4dc30aad4bc9cc89c00b8e2e
    Merge: 3bb1217 4cab7f0
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Feb 11 15:24:54 2013 -0600

        Merge branch 'vocms236-2' into 2013-2-11

        Conflicts:
        	src/couchapps/WMStats/_attachments/js/minified/global.min.js

    commit df683cf567bddde99e95e47d21ebefeacd18c065
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Feb 11 15:20:43 2013 -0600

        add agent detail info

    commit 4cab7f09ce339ab3f776106a09e6e259bb61ca47
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Mon Feb 11 15:17:16 2013 -0600

        add link to the job number

    commit 4d4f96014232ef9c43abce5aafbb2aaaae715a5e
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Wed Feb 6 14:10:38 2013 -0600

        add agent detail work in progress

    commit 9dc747372a1ed10d12b78ad8a764edda3ca0dd5c
    Author: Seangchan Ryu <sryu@fnal.gov>
    Date:   Fri Jan 25 15:23:38 2013 -0600

        add queue column

commit c107dcc406631f543dac1b08fae275f3d348b87f
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date:   Wed Feb 20 17:18:39 2013 +0100

    fix bug in stageout exception handler

commit 4821294c30001d0a51c2d05c00b131979485721c
Author: Manuel Giffels <giffels@gmail.com>
Date:   Wed Feb 20 13:51:57 2013 +0100

    SiteDB emulator

commit 0a8a3f3e506443fa764cd0097555cae6cb19ea62
Author: Manuel Giffels <giffels@gmail.com>
Date:   Wed Feb 13 15:32:33 2013 +0100

    Move Service/SiteDB to SiteDBv2

commit b0996c4ff8a096ff4c78c3e2b3d4e39b4bde6311
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Wed Feb 20 08:31:32 2013 -0600

    0.9.43

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 5b6c4ed3a2fffd32e96d82648105efebfee817d6
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Wed Feb 20 08:31:02 2013 -0600

    0.9.42

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit b1027726b443c107d13882cc24bd777e1e3f79ef
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Tue Feb 19 17:42:54 2013 -0600

    Improve ACDC splitting in global queue

    Allow two modes of operation dependant on the splitting algorithm
    that will be used for the top level task. One is the currently used
    mode that splits in fixed sized chunks and the other one is a single
    chunk mode that prevents blocking the local queue when splitting
    algorithms required wmbs injection before creating jobs.

    Also remove the cleanup tasks from merge ACDCs, as it stands now
    these won't work and are currently removed manually by ops through
    hacky scripts.

    Fixes #4043
    Fixes #4029 (Not properly but it does fix it)

commit 41e1cbf496f0f096d20d7e407a5f098a25760cb5
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Tue Feb 19 13:11:51 2013 -0600

    0.9.41

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 33cf381135143a33e3b3505fccedf9e74c418583
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Tue Feb 19 15:00:49 2013 +0100

    ReqMgr databases consistency checks and cleanup

    ReqMgr prod databases (CouchDB CMS web couchdb/reqmgr_workload_cache and
    Oracle)

    There were 498 requests in the CouchDB which were not present in Oracle.
    Probably failed during injection and since ReqMgr injection is not implemented
    as an atomic operation a request was put in CouchDB but not in Oracle.
    Checked with Edgar, requests in question were not tracked in GlobalMonitor and
    were all agreed to be deleted from CouchDB. Also, since some CouchDB document
    fields are not properly filled (#4380) by ReqMgr (e.g. RequestStatus), it's
    not possible to tell more about those requests just by ReqMgr itself.

    Total 498 requests deleted from CouchDB.

    This patch is part of #4388

commit 4391a8a138306b0b1458d2e6f2f1eb8296d6d138
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 18 14:17:31 2013 -0600

    Fix unit tests

    A few tests went down with the latest wave of commits, nothing serious.
    This fixes the tests, no changes to functionality.

commit ddf7eb711dc6bf4692c8cdedd402f3ee231861b9
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Sun Jan 6 22:15:46 2013 -0600

    Open requests feature

    Allows a request with an input dataset to continue acquiring and processing new
    blocks after assignment. It works by continuously polling DBS for new blocks in the
    dataset, and when found these are splitted as usual and queued in the global queue.

    It changes the ReqMgr running state to running-open and running-closed, where
    running-open means that the request is still looking for new blocks. running-closed
    is equivalent to the old running state.

    The criteria to move a request from running-open to running-closed depends on a predefined
    timeout, which comes with the request schema. If not given the request will be closed
    as soon as it starts running. The request can also be closed at any time
    through the web interface or the REST API. Once a request is closed, it will continue
    process the already acquired inputs and will stop polling for blocks.

    The timeout counts from the last time a block was found (either closed or open).
    This feature can only be used with workflows which take a dataset as input,
    namely ReDigi, ReReco, MontecarloFromGEN.

    Finally, this also fixes the reported issue when a single open block in a dataset
    can prevent the request from being acquired. The new behavior is to ignore open blocks
    in the initial splitting, and the request will be acquired as long as there is at least
    one closed blocks. The continuous polling will only acquire closed blocks.

    Also zero event blocks with no site will be ignored, all these ignored blocks will be accounted
    on the RejectedInputs field of the global workqueue element.

    Fixes #4400
    Fixes #4373

commit d4b80f29f5d510e7e1fbdb699774bfd2082956fa
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 18 02:55:34 2013 -0600

    Expose AutoApprove sites to ReqMgr

    Also allow modification of move/replica subscriptions for custodial
    and give a more descriptive parameter to the priority option.

    Fixes #4356

commit df75ecc3a959f966388c6ca7206a0e01299f93b9
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 18 00:40:41 2013 -0600

    Fix JSON missing lumi function

    It was broken due to improper testing and old couchdb version.

    Fixes #4338

commit a96e484c34ebf735a391c5542ed600b4b2ab1460
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Mon Feb 18 00:11:53 2013 -0600

    Fix display of output datasets

    Datasets produced by output modules that are never stored in merged
    area were still reported by the listOutputModules function of WMWorkload.
    Fix that by adding a flag that clearly marks transient and permanent
    output.

    Fixes #4422

commit fe8229e8ddce13c90c6d5d1a3f345d04eca76eb1
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Sun Feb 17 21:39:49 2013 -0600

    Fix splitting page issues

    Duplicate argument keys cause the object returned to the reqmgr
    to be an array. Make sure that there are no duplicate keys,
    also display the correct values for EventAwareLumiBased splitting.
    Also fix modification of max time for merge.

    Fixes #4409

commit 865bd8846dd41d42dc1b1ea2cce567bedccc4bf5
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Feb 15 10:38:39 2013 -0600

    Fix deletion of couch documents

    Delete the fields in the document before commiting with _deleted = True,
    make sure that WorkQueue replication replicates the deletion but avoids
    creating deleted documents from other queues. Ensure the same in
    WMStats.

commit 47c4c52c0cd80321a2ec60170c2ac60efcbad755
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Fri Feb 15 17:09:57 2013 +0100

    ReqMgr delete request from CouchDB

    Delete request when called deleteRequest() method from Oracle
    as well as from CouchDB.
    Passes all ReqMgr tests locally, however doens't work because
    of a bug in CouchDB low level document delete: #4403 originating
    from #4396 - once this low level problem is resolved, it
    shall be fine. Tests added into reqmgr.py

    Fixes #4289

commit 2e087c240831160acbe5be6a84224750743fc4cd
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Thu Feb 14 16:28:43 2013 +0100

    ReqMgr - Processing, Production dead code clean-up

    Entire two packages

    WMCore/RequestManager/RequestMaker/Processing
    WMCore/RequestManager/RequestMaker/Production

    were unused, and just one class referenced from the unittests,
    and even that was in vain.

    Removing all files, tested on VM with reqmgr.py
    Fixes #4420

commit a7a958b057891d0f6ec50980ce8f6246aeca3bf6
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Thu Feb 14 14:04:36 2013 +0100

    ReqMgr - Fix for ACDC-Resubmission Campaign, etc

    -Added request state transition from completed to rejected (last agreement,
        the previous was to aborted).
        Fixes #4413
    -ACDC (Resubmission) requests should inherit the Campain field.
        from the original request.
        Fixes #4378
    -reqmgr.py script tests Resubmission requests.
    -clean-up of some unsed import statements.

commit 446eb3682feda096a06085995d9c16f7070e9a60
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Thu Feb 14 14:03:09 2013 +0100

    ReqMgr - further Ops issues, fixes

    -added access control when deleting requests
    -Added request state transition from completed to aborted.
        Fixes #4413
    -Merged LFNs bases on the main assign page as well as on the
        assign/one are now the same offering complete list of bases
        regardless of the request type which was the case on the
        assign/one page.
        Fixes #4343

commit f87c8e900a45788514055e455efe1ac7c541e4e6
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Thu Feb 14 14:01:34 2013 +0100

    Change SofTimeout default parameter value

    SoftTimeout workflow parameter value changed from 167000 to 129600.
    Fixes #4411

commit d2e2e920eb28f9720d2fbd7fc72ab31b4f637d79
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Wed Feb 13 02:09:10 2013 -0600

    Don't change file status while in upload only mode.

commit 37924116dc239102f07ee229ea7cffda3497c9fa
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Tue Feb 12 11:07:25 2013 -0600

    make rejected workflow to be archived and remove from tracking in wmstats

commit 7ad1da0dfb0795e5a701b86a7762221785bfd901
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date:   Mon Feb 11 15:18:44 2013 +0100

    fix bug in file deletion in CERN stageout plugin

    The last commit introduced a bug in the CERN stageout plugin that
    made file deletions fail. Fixed that and also restructured the
    DeleteMgr class to not use and exception handler as a flow
    control mechanism. Also added a printout so some possible
    error messages aren't supressed anymore.

commit cc60e96fc85556737f4ee8711801b5fda139110c
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Fri Feb 8 13:42:02 2013 -0600

    Bugfix Smart error handler

    Bugfix for extreme cases of null reports and also
    a few cases where jobs could get stuck in jobfailed.

commit 7110dcc94f7bac8bcd2bc6729cca526475080940
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Fri Feb 8 10:47:37 2013 -0600

    Fixes for problems uncovered during the DBS2/DBS3 consistency check:
      - Block size and file count is now correct.
      - Physics group is not correct.

    This also adds a legitimate unit test which validates all data injected
    and tests the special T0 DBS2/DBS3 upload only mode.

commit 0fe3aa5b195d48c1789397c8345a51eb1514b485
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Fri Feb 8 10:14:14 2013 -0600

    Fixing delete stuff

commit 84814f847a7f2301bc461ac46208d9785d844f91
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Fri Feb 8 15:25:46 2013 +0100

    Enforce integer processing versions

    This extends the support in WMWorkload and WMTask to include a
    processing string along with the processing version and acquisition era.
    The processing version is now enforced to be an integer and the lfns
    are built as following:

    <acquisitionEra>-<processingString>-v<processingVersion>

    The job reports contain all these three elements for each output file
    however the DBSBuffer will only use the processing version and acquisition
    era.

    The ReqMgr changes to allow the specification of these parameters in the
    assign page, also updated the tests and templates. Include a test template
    for TaskChain with multiple tasks, multiple acquisition eras and versions.

    Fixes #4007

    - added assert into reqmgr.py to check for correct request state
    - modified template values of AcquisitionEra parameter

commit ca083d6f7b0797e5a7f4df6e39084e8451996550
Author: Andrew Melo <andrew.melo@gmail.com>
Date:   Wed Feb 6 09:24:12 2013 -0600

    Fix how we delete documents in couchdb

    A weird hitch in couch's delete semantics is that if you don't use the
    DELETE verb (i.e. you commit with _bulk_docs and set _deleted=true), it
    stores the whole data in the delete tombstone. See:

    https://issues.apache.org/jira/browse/COUCHDB-1141

    Since (AFAIK), we don't ever use _deleted_revs=true to look at things,
    it means that we have tons of old data sitting around in couch that's
    basically invisible. This patch strips the document before comitting it
    so the only thing lefts are _id, _rev and _deleted

commit 893ed535d9c9418896b683d0dffbb261ac8019ac
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Feb 6 15:13:54 2013 -0600

    prevent task archiver crashing when central couch server is down

commit 826bbf61eb1b96e061d3f7f73db5905e174aab47
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date:   Wed Feb 6 17:46:41 2013 +0100

    add support to stageout from HLT Cloud at CERN

    The VMs on the HLT Cloud at CERN don't have access to the eos environment.
    Change the stageout plugin to only use generic xrootd commands.

    Patch was developed by Stephen Gowdy, I did test it in a replay though.

commit 4d966870f04e34a39eec91d02c773aa7a4c5157f
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Feb 6 12:18:49 2013 +0100

    ReqMgr localhost testing, running

commit 92364a9793832d7c2b71737cc25c097e7f11d201
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Jan 24 17:48:21 2013 -0600

    Fix a typo in the CleanCouchPoller code

    It can cause unexpected errors, when an error occurs

commit ecd713ff43a434298d4333bc17dc3c866a4219f4
Author: Zdenek Maxa <zdenek.maxa@cern.ch>
Date:   Tue Feb 5 12:10:28 2013 +0100

    0.9.40

    Signed-off-by: Zdenek Maxa <zdenek.maxa@cern.ch>

commit a8280d4cbe012835f58e8e20f175291944098de2
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Mon Feb 4 15:20:23 2013 -0600

    Fixes from the testing of DBS3 upload.

commit 365be2f9bb7c24ffe8f73d0b25928860720c4b9c
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Fri Feb 1 13:15:33 2013 -0600

    add script to unregister agent when agent is retired or inactive

commit c754fc3f6dfb9525b758559d81c9dca3c5f3081d
Author: Seangchan Ryu <sryu@fnal.gov>
Date:   Wed Jan 30 15:26:38 2013 -0600

    check components are alive using pid and also thread status

commit 125a8e1a7f711aefab6506f1eecaef709c42a184
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Jan 31 18:03:21 2013 -0600

    Fix unit tests breakage due to Specs_t

    Make sure that all unit tests in Specs_t use the temp directory
    to store sandboxes and emulators for DBS

    Fixes #4363

commit 50814a18000e49f2b1141ff23b73c40c12dde2bd
Author: Diego Ballesteros <dballest@fnal.gov>
Date:   Thu Jan 31 10:23:54 2013 -0600

    Get the correct times out of a report if any

    The function to retrieve the start and end time out of a FJR had a
    bug where a None time would make it return always a None stopTime.

    Fix it and also don't allow the ErrorHandler to get stuck if
    the jobs don't have running time information.

commit 178edd255d9d5465000ecc67a22e5a333c97c881
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Jan 30 16:43:32 2013 +0100

    ReqMgr testing enhanced by TaskChain requests

commit 638ee4c6f4c8e3c3730d558c34485d0cb204552c
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Wed Jan 30 16:30:02 2013 +0100

    ReqMgr CustodialSite bug fix.

    CustodialSite is an argument not used for long. (Non)CustodialSites have
    superseded this, however no one bothered to remove all dead functionality and
    parameter definition. On cloned requests, the default value (None) of
    CustodialSite request parameter appared and PhEDExInjector was failing on
    that. This patch removes CustodialSite from ReqMgr only.

commit ffdd78bbbbc4cd0d69dd5dd0377e402f2445b6b1
Author: Stephen Foulkes <sfoulkes@fnal.gov>
Date:   Tue Jan 29 16:23:38 2013 -0600

    0.9.39

    Signed-off-by: Stephen Foulkes <sfoulkes@fnal.gov>

commit 869b1401aa8de518ddcbadfb69922ca714f2ad77
Author: Zdenek Maxa <zdenek.maxa@hep.caltech.edu>
Date:   Mon Jan 28 17:59:41 2013 +0100

    ReqMgr cloned requests priority fix

    When cloning a request which had changed priority, the newly cloned request
    priority is on default values. Fixes goes via MakeRequest.createRequest() to
    database DAOs Request/New.py
    Also reqmgr.py checks the case.

    Modified default value SoftTimeout on Ops request.

commit 0a54583f6fceee795a11002de0ce7ae30300a3b2
Author: Dirk Hufnagel <Dirk.Hufnagel@cern.ch>
Date:   Mon Jan 28 16:44:44 2013 +0100

    fix cmsrun config for endofrun dqm harvesting

    EndOfRun DQM harvest jobs need a special cmsRun configuration.