shadow/shadow — pull request #44 by caffeineshock (2012-06-18 12:41:18+00:00)
Title: Plugin to simulate a browser

Purpose
======

To examine the effectiveness of and countermeasures against [traffic analysis attacks](https://en.wikipedia.org/wiki/Traffic_analysis) on the Tor network with Shadow/Scallion one needs to be able to access virtual servers in the way most users use Tor to access websites. Thereto I implemented this plugin, which tries to immitate the behaviour of a modern browser: It not only downloads an HTML document (which is already possible with the filetransfer plugin), but also parses the HTML and downloads the following embedded objects:

+ favorite icons
+ stylesheets (embedded with a link-tag)
+ images (embedded with an img-tag)

Implementation
===========

Like a browser the plugin opens multiple persistent HTTP connections per host. The maximum of concurrent connections per host can be limited though (Which all modern browser do [as well](http://www.browserscope.org/?category=network)). When there are more downloads than connections available the connections are reused once a download finishes.

To implement this in shadow I used the filegetter-functions from the filetransfer-plugin. I needed to extend those functions a little bit though:

1. If `save_to_memory` is set to `TRUE` in `filegetter_filespec_s` then the content of the requested file is written to `GString* content` in `filegetter_s`. I need this to be able to parse the HTML document downloaded initially.

2. To manage persistent connections the flag `persistent` was added to `filegetter_serverspec_s`. When `persistent` is set, a connection is not closed when the download is complete (state `FG_CHECK_DOWNLOAD`). This means if `persistent` is set, the connection is not closed until `filegetter_disconnect` is explicitly called by the user and that multiple files can be downloaded through the same connection

Example Usage
============

To simulate browser requests to www.wikipedia.org the site has to be mirrored to the local filesystem:

```bash
wget -H -p http://www.wikipedia.org
```

This will create directories for each hostname (namely `bits.wikimedia.org`,`en.wikipedia.org`, `upload.wikimedia.org` and `www.wikipedia.org`) and download the HTML document and the embedded resources to the respective directory.

Then each hostname need to node in shadow. The host file for this example should therefore look something like this:

```xml

```

The arguments for the browser plugin denote the following:

1. HTTP server address/name
2. HTTP server port
3. SOCKS proxy address/name
4. SOCKS proxy port
5. Maximum amount of concurrent connection per host
6. Path of the HTML document

The output should be like the following:

```
[browser_start] Trying to simulate browser access to /index.html on www.wikipedia.org
[browser-message] [client.node-23.0.0.0] [browser_completed_download] first document downloaded and parsed, now getting 15 additional objects...
[browser-message] [client.node-23.0.0.0] [browser_activate] done downloading embedded files
```

To Do
====

- Printing usage
- Writing a Readme (This might be a good start ;-))
- Printing useful statistics like aggregated downloaded and upload bytes etc.