internetarchive/wayback — pull request #12 by ikreymer (2014-02-22 23:13:40+00:00)
Title: Robots parsing fix

Fix robots parsing logic so that blank lines (or any non User-agent) lines between User-agent lines cause rules to be reset.

That is,

User-Agent: A

User-Agent: B
Disallow: /

disallow rule applies only to B

User-Agent: A
User-agent: B
Disallow: /

disallow rule applies to both A and B