chfoo/wpull — pull request #169 by mback2k (2014-08-21 16:02:07+00:00)
Title: New hooks to be notified about URLs being added or retrieved from the queue

This merge request implements new hooks in order to be notified about URLs being added or retrieved from the queue. There is still some room for improvement since currently the list of added url_infos has to be recreated in item.py since only the URLs themselves are passed to database.py.

> <+mback2k> chfoo: here is my first draft for new hooks: https://github.com/mback2k/wpull/commit/a73092dd531dbcba342f5587a28ae9100e391196
> <+mback2k> chfoo: my only problem is that the URL-table performs a unique-operation on URLs and therefore multiple adds of the same URL result in at most one dequeue operation.
> <+mback2k> chfoo: that makes it impossible to implement a proper counter unless the add-operation returns wether the URL already existed.
> <@chfoo> chfoo: the new hooks are looking good. yeah, unfortunately i don't check whether the url already exists for performance reasons.
> <+mback2k> chfoo: maybe the SQL statement in https://github.com/mback2k/wpull/blob/master/wpull/database.py#L250 could be adapted to return a different result code on INSERT if already exists?
> <+mback2k> chfoo: I mean, the unique-constraint in the database has to make the check anyway, so there should be a quick way to get that information
> <+mback2k> without an additional query, of course
> <+mback2k> chfoo: I will take a look
> <+mback2k> chfoo: I found a way to get it working without any additional SQL queries :)
> <+mback2k> chfoo: some more progress: https://github.com/mback2k/wpull/commits/topic/queue_hooks