dmwm/CRABServer — pull request #4536 by AndresTanasijczuk (2014-11-25 23:31:46+00:00)
Title: Rewrite cmscp and PostJob with functionality improvements.

While I started to work on improving some functionality of the stage out algorithm (I describe these improvements below), I decided I will rewrite and reorganize parts of the cmscp and PostJob codes, making them more readable (IMO) thinking about future persons that will need to read and learn about the code. I added lot of comments, changed names of variables and functions to be more representative of what the mean and do, and also took into account writing the code so that pylint has less complains. I also tried to improve the messages in the job and postjob logs.
The cmscp code is now written in a more manifest step-by-step way, so that it is clearer what comes first and what comes after. And for each step there is a condition that needs to be satisfied in order to execute the step. We also have an exit code for each step which is used in the conditions (e.g. execute this step if these other steps were successful). At the end of each step update the global cmscp exit code if necessary. Each step can be skipped by turning-on a corresponding flag in the code (helpful for debugging).

Now the list of functionality improvements:
1) Inject transfer requests to ASO database only after all local transfers are successful, as opposed to inject per file. (Bulk injection is not done yet.)
2) Stage out all files using the same transfer policy (either local or remote) instead of doing it per file. This is, if the transfer policy in the TaskWorker is "local,remote" (meaning first try local stage out and if it fails try remote stage out), we first try to do local stage out for all files, and if one local transfer fails, we clean the local temporary storage (leaving the log file, because want it to be available for the user) and try remote transfers.
3) Upload log file metadata from WN. With this feature users are able to retrieve the log file with the 'crab getlog' command even if the post-job doesn't run (it was happening until now that if the job failed, the post-job was not getting up to the point of uploading the log file metadata, and the user could not retrieve it with 'crab getlog').
4) Add cmscp exit code to the job report.
5) Add 'rest_host', 'rest_uri', 'source_lfn' (this is the same as the current 'lfn') and 'destination_lfn' to the ASO documents.

This PR fixes the following issues: #4110, #4420, #4454, #4465, #4507.

I have tested the cmscp code extensively, but not that much the PostJob. We agreed that Justas will do more tests in the testbed.