Byzantium/Byzantium — Issue #83: Distributed search engine.
Author: virtadpt
Created: 2012-01-18 15:21:40+00:00
State: open
Labels: Feature request, Request for comments

A potential problem of meshes over a certain size is finding information. A search engine of some kind would be necessary to facilitate this. Given our use cases, however, reaching an external search engine (or allowing external search engines to access a mesh) is not feasible.

I've been experimenting with YaCy (http://yacy.de/en/), a peer-to-peer distributed search engine that spiders content proactively as well as proxying and spidering content that users browse (note: YaCy is designed with privacy in mind, so by default it detects and skips over things like webmail and private messaging pages). It would be possible to configure YaCy to spider and index things that users of a mesh look at while ignoring sensitive information (specified with regular expressions applied to URIs).

The downsides of this are that it requires Java, it requires a fair amount of RAM (my home node uses about 600MB of 4096MB), is fairly CPU intensive (resulting in faster battery drain), and is fairly bandwidth intensive (so we'd have to be careful about how aggressively nodes spider and how many levels of links they pull to index).