OpenRefine/OpenRefine — pull request #857 by FWennerdahl (2014-02-14 10:05:15+00:00)
Title: Refactoring of project data storage

This refactorization centralizes the access to all project files (data, meta and history) in a ProjectDataStore instance tied to each Project. Following the design of a (partially) abstract ProjectManager and an implementing FileProjectManager, we created an abstract ProjectDataStore class and a FileProjectDataStore implementation. The ProjectManager now contains an abstract factory method for serving a ProjectDataStore instance.

Our reason for making this refactorization was that we identified several race conditions where file operations would fail due to the lack of thread-safety. Most notable was that deleting projects could fail if another operation was carried out on the files by another thread. This caused files for deleted projects to remain on disk. In our use-case this happened for approximately 15% of the projects.

To remedy this, the FileProjectDataStore uses a ReentrantReadWriteLock to assure thread safety as long as the same Project instance is accessed. In some cases new Projects are created to access files, however in these cases the project does not exist in the ProjectManager cache such as when starting up, recovering and importing projects.

Test cases were updated where needed. We decided to preserve the ProjectManager interface for project operations (such as loadProject, saveProject etc.) to not disrupt the tests too much. The FileProjectManager methods simply call the equivalent methods in Project so these implementations could be moved to the base class since they will be the same for any implementation of ProjectManager. Alternatively, these methods could be removed from the ProjectManager interface altogether and called directly on Project in a later refactoring. We'll leave this decision up to the OpenRefine team.

We are aware that this is a huge pull request but given the task we didn't see a feasible way to divide it into multiple requests.