spotify/luigi — pull request #270 by freider (2014-02-07 15:32:41+00:00)
Title: [postgres] Adding exists query to reinforce completion logic

Set an optional exists_query method on a CopyToTable subtask to use as a backup in case the marker table default mechanism determines a dataset doesn't exist. This is to prevent duplicated data insertion.
If exists_query determines data to exist, the update marker is reinserted to "fix" the cache.

The reason exists_query isn't used within the complete() method is to avoid heavy polling of the destination tables with possibly complex completion queries. With this patch, the exists_query will only be executed if
1. there is no completion marker for the data set in the marker table
2. AND input data to the task exists, so the run method is triggered

If both of those conditions are met, but the exists query decides that data actually exists it will not be reinserted, but instead a new marker will be put in the marker table (the output target will be `touch()`ed.

In the future as this module is developed, I hope we can add more kinds of queries to postgres data sets. For example select_query and delete_query, letting people in an easy way parametrize reading and deleting the data from a dataset in subsequent tasks or tools.