deluge-torrent/deluge — pull request #6 by bendikro (2014-08-03 21:03:33+00:00)
Title: Send only updated/changed torrents to the client

This branch contains the following modifications:

* Client can request to get only the torrents that were updated since last time
* This reduces the amount of network data that needs to be transfered between the client and server greatly. With thousands of torrents, the overhead of processing the entire torrent list to be sent and received regularly causes the major part of the CPU usage as seen by profiling.
* The filtering of the torrents has been split between the server and client. With thousands of torrents, the workload of filtering the torrents each time the list is sent becomes a problem. This has been changed so that the server only filters on the owner to make sure only the torrents the user is allowed to fetch are returned. The rest of the filtering is performed on the client side.
* Much faster to change the list on the client side as it no longer has to ask the server.
* The filtering functionality is implemented with PyDbLite.
* This should make it possible to implement advanced filtering quite easily.
* The filtering is much more efficient than the current code

It worked the last time I tested it, but it requires some more work.
- [ ] Write more unit tests
- [ ] The code changes for the webui should work but the code should be reviewed (and probably rewritten by someone who knows and likes javascript)

Feedback is very appreciated. :wink: @gazpachoking

Note: PyDbLite is a separate project, however I've made loads of changes to the code which I plan to get released on the official project.

# Changes on the server side
### deluge/core/core.py
Increased alert_queue_size to 20000, we should find a better solution than hardcoding this.

Changed functions:
* **get_torrents_status:**
* Added keyword argument **only_updated** which makes **get_torrents_status** only return the torrents that have been updated since the last call.
* Argument **filter_dict** is now obsolete.
* Argument **diff** is now obsolete.
Removed functions:
* **get_filter_tree** - The filtertree is now created on the client side

### deluge/core/filtermanager.py
Most of the functionality of filtermanager.py has been moved to the client side (torrentfilter.py). filtermanager.py now only filters on the owner so the client will only receive the torrents it is authorized to see.
It filters on owner using the new databases in deluge/include/PyDbLite.

### deluge/core/rpcserver.py
Changed functions:
* **connectionLost**
* Now emits ClientDisconnectedEvent when client disconnects

### deluge/core/torrent.py
Changed functions:
* **set_tracker_status**
Now emits TorrentTrackerStatusChangedEvent when torrent status is set

### deluge/core/torrentmanager.py
Added **status_dicts** which contains all the torrents for each session. When a torrent is updated (either by libtorrent or a plugin) it is removed from the cache to mark it as updated.

Changed functions:
* **handle_torrents_status_callback**
* Creates a status based on the **only_updated argument**, so if **only_updated** is True, only the torrents that do not exist in the cache (for this session_id) are returned.
* **on_alert_state_update**
* Removes the updated torrents from the cache, and emits TorrentsUpdatedEvent

New Functions:
* **update_torrent_cache(torrent_ids)**
* Removes the **torrent_ids** from the cache

# Changes on the client side

### deluge/ui/gtkui/addtorrentdialog.py
Changed functions:
* **add_from_files**
* Added check on duplicate torrents so that the popup “Duplicate Torrent(s)” is only shown once.

### deluge/ui/gtkui/torrentview.py
Changed to now only update the necessary rows.
The main difference is that **self.status** now always contains all the torrents, not only the torrents that are displayed. Which torrents that are currently displayed in stored in the set self.visible_torrents which is updated in **_on_get_torrents_status**.

Changed functions:
* **_on_get_torrents_status**
* Updates self.visible_torrents no contain the currently visible rows.
* **update_view**
* Added arguments:
* **only_updated_ids**
* The torrent_ids that have been updated with new data. If None, all have been updated.
* **to_show**
* The **torrent_ids** to show (change filter column value to True)
* **to_hide**
* The **torrent_ids** to hide (change filter column value to False)

Removed functions:
* on_torrentstatechanged_event
* No longer needed as this is handled by sessionproxy.
* **mark_dirty**
* The dirty functionality was not used anymore (since the last major torrentview changes) so it was completely removed.

### deluge/ui/sessionproxy.py
Has been completely changed. It no longer keeps track of cache times for each value. This was extremely expensive, and is no longer needed as the torrents that have been updated on the server side are now updated automatically.
**self.updated_ids** contains the torrent ids that have been updated from core and not yet updated in the ui.

All of the filtering previously performed in filtermanager is now performed in by **self.torrentfilter**. This reduces the workload for the daemon tremendously and makes it much faster on the client side to get the filtered results.

Changed functions:
* **get_torrents_status**
* The biggest change is that it now takes the status dictionary as input and updates the status with the torrents as needed. It returns which torrents were updated and which torrents that should be shown or hidden in the view.

New arguments:
* **only_updated=False**
* Give only the updated torrents
* **from_cache=True**
* If it should wait for the data from core, or return data from the cache.
* **status=None**
* The status dictionary containing all the torrents and torrent data
* **current_ids=None**
* The currently displayed torrent ids.

### deluge/ui/torrentfilter.py
This does all the filtering on the client side which was previously done in filtermanager.py on the server side.
torrentfilter.py uses PyDbLite (which uses SQLite if available, else pure Python) to perform all the filtering. The filtering code should be cleaner and simpler than the old code, and also _much_ for efficient (with many torrents).

### deluge/plugins/Label/deluge/plugins/label/{core.py,gktui/__init__.py}
Changed to work with the new filtering

Changed to handle “hide owners” in sidebar:
deluge/ui/gtkui/filtertreeview.py
deluge/ui/gtkui/glade/main_window.ui
deluge/ui/gtkui/menubar.py
deluge/ui/web/server.py
deluge/ui/gtkui/gtkui.py

# Changes for both client and server
### deluge/event.py
* Added events
* TorrentsUpdatedEvent
* TorrentTrackerStatusChangedEvent
* ClientDisconnectedEvent

### deluge/common.py
Added functions:
* **total_size**: Calulcates the memory usage of an object

## PyDbLite for efficient filtering

### deluge/include/PyDbLite/
Added files:
* DelugeDB.py
* Contains Classes to create and setup the PyDbLite database
* PyDbLite.py
* Contains pure Python implementation of a “database”
* SQLite.py
* SQLite wrapper
* SQLiteExtended.py
* Modifications/Extensions to SQLite.py