crawljax/crawljax — pull request #447 by insighti (2014-12-01 02:38:37+00:00)
Title: avoid URISyntaxException thrown on broken links

Hey there,

there's a problem we've run into while crawling a ajax-heavy page. We do understand that it is more of a page problem rather then crawljax's, but we still think it might come handy to fix this in crawljax.

While crawling the page, crawljax runs into a (by RFC definition) broken URL - eg. `https://hostname.tld/#something=blah|somethingelse=blah1`

Pipe `|` is not an allowed character in URL Fragment definition in [RFC 3986](http://tools.ietf.org/html/rfc3986#page-24).

However as it crashes like this:

```
java.lang.IllegalArgumentException: Illegal character in fragment at index 36: https://hostname.tld/#something=blah|somethingelse=blah1
at java.net.URI.create(URI.java:852) ~[na:1.8.0_20]
at com.crawljax.util.UrlUtils.isSameDomain(UrlUtils.java:82) ~[crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.Crawler.crawlerLeftDomain(Crawler.java:407) ~[crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.Crawler.crawlThroughActions(Crawler.java:325) ~[crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.Crawler.execute(Crawler.java:124) ~[crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.CrawlTaskConsumer.handleTask(CrawlTaskConsumer.java:75) [crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.CrawlTaskConsumer.pollAndHandleCrawlTasks(CrawlTaskConsumer.java:67) [crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.CrawlTaskConsumer.call(CrawlTaskConsumer.java:47) [crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at com.crawljax.core.CrawlTaskConsumer.call(CrawlTaskConsumer.java:16) [crawljaxRunner-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_20]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_20]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
Caused by: java.net.URISyntaxException: Illegal character in fragment at index 36: https://hostname.tld/#something=blah|somethingelse=blah1
at java.net.URI$Parser.fail(URI.java:2848) ~[na:1.8.0_20]
at java.net.URI$Parser.checkChars(URI.java:3021) ~[na:1.8.0_20]
at java.net.URI$Parser.parse(URI.java:3067) ~[na:1.8.0_20]
at java.net.URI.(URI.java:588) ~[na:1.8.0_20]
at java.net.URI.create(URI.java:850) ~[na:1.8.0_20]
... 12 common frames omitted
```

It'd be a shame to have crawljax break every time it hits such URL as it only needs to compare hostname anyway - fragment is totally irrelevant.

Attached is a proposal of a quick fix using an internal method to extract only the relevant part of the URL before creating URI object.

Cheers,
insighti team