qiime/qiime_web_app — pull request #63 by douginator2000 (2012-11-13 17:49:59+00:00)
Title: Discovered a further issue with the job delete synchronization code. 

This hopefully contains a final fix for this issue. The logic is as follows:

Since these are completely separate processes on different machines, there aren't too many ways for each process to communicate with each other. The original fix utilized a synchronization file to tell the other proceses that one process was already performing the delete operation. The other processes that saw this blocked until this file was removed.

Unfortunately this didn't quite cover it when the delete operations were extremely fast - it was still possible for one job to completely finish it's delete, cleanup the sync file, then have another job get started after the fact and perform the delete all over again.

The new solution is this: A cleanup_in_progress file is created by the first process that performs the cleanup operation. This process then performs the deletion of database data and file system data. When it completes, it writes yet another file called cleanup_complete. The other processes check first to see if the cleanup_in_progress file exists. If so, it will wait until the cleanup_complete file exists, at which time they simply skip the delete operations and proceed with processing.

Hopefully this logic guarantees that only one process will ever enter that code section and the other processes must wait until that proces has finished deleting.