tricycle/predictionbook — Issue #22: Narrow site results in Google
Author: gwern
Created: 2011-12-15 19:16:55+00:00
State: open

I was quickly pasting a link summarizing existing predictions for a fine point in _Methods of Rationality_ (https://encrypted.google.com/search?num=100&q=hat%20and%20cloak%20site%3Apredictionbook.com) and I noticed, not for the first time, that the actual prediction was being drowned out by the user pages.

Inasmuch as you can get to any relevant user page by the actual prediction, I think they're noise.

Fortunately, there's a very easy solution - we can just exclude /users/ in the robots.txt. However, in keeping with my usual [long-term interest](http://www.gwern.net/Archiving%20URLs), I still want the Internet Archive to have access; I think we can combine the two like thus (wrote it up based on the Wikipedia info and http://www.archive.org/about/exclude.php and then I found a similar question at http://www.archive.org/post/234741/robotstxt ):

```
User-agent: ia_archiver
Allow: /users/

User-Agent: *
Disallow: /users/
```

The downside of this approach is that someone searching for a particular user would see their user-page pop up, but rather every prediction they've been involved in. This isn't a disadvantage for me, but it may be for others.

A less invasive but also less useful approach would be to augment the built-in Google site search with `-site:predictionbook.com/users/`. (Not as useful for me, since I rarely launch my PB site searches from that dialogue, but outside Firefox entirely.)

Anyway, I'm going to patch robots.txt as suggested above.