saga-project/saga-python — pull request #199 by chris-zen (2013-07-05 10:21:50+00:00)
Title: Fix #167: _retrieve_job uses qacct to get info about jobs not in the qstat queue

Hi, I send a pull request with the modifications made to the SGE adaptor to support retrieving job information for jobs that go out of the qstat queue using accounting information.

The SGE CLI interface is the worst I have seen ever and have some issues. One of them, that I can not understand, is that when you run *qstat -j job_id* you don't get the state of the job nor the start_time. To get this information you have to run another *qstat* process without *-j* and parse the output. Other issue is that when a job finishes/fails it goes out immediately from the *qstat* list. Here there are two possibilities: If accounting information is not activated then we don't have any chance to get the information unless a wise mystic inquire. In the lucky case that it is activated then there is a gap in time between the job goes out of *qstat* queue and when it is available through *qacct* (sometimes few seconds, other times more than 10 seconds in my virtual machine). I implemented the retrieval of job information through *qacct* in a separate function ( *__job_info_from_accounting* ). There, there is a maximum number of 10 trials with lapses of 1 second before a None is returned (and consequently an exception is raised). This makes some job operations quite slow and prone to error when the job limbo time exceeds 10 seconds approx. It is possible to know whether accounting is activated or not (using *qconf -sconf | grep accounting=true*) and avoid the *qacct* part in case it is not but I have not implemented it.

Other way to address this issue would be to take the approach proposed in #180 and maintain a remote process open all the time acting like a server and using *DRMAA* to send jobs, get status, ... It would be a bit complicated, and in the case that you loose the *DRMAA* session (network cut, ...) when reconnecting you would have lost those jobs that finished while *DRMAA* process was off ( *DRMAA* 1.0 doesn't have session persistence guaranteed). Furthermore, IT's restrict the maximum time per process in the submission host usually.

So, no other solution comes to my mind apart from waiting for *DRMAA* 2.0 implementation or Open Grid Scheduler team takes care of fixing CLI interface (which I don't think).

Sorry for my hard words about SGE but was frustrating trying to find a solution with which I felt comfortable enough.