scrapy/scrapy — Issue #82: SSL compatibility issues with some servers
Author: johtso
Created: 2012-01-25 19:48:48+00:00
State: open

Attempting to fetch [this page](https://rn.ftc.gov/pls/textilern/wrnquery$.startup) over HTTPS resulted in the not very enlightening error `Connection was closed cleanly.`

```
ERROR: Error downloading : Connection was closed cleanly.
```

Some very helpful people on #twisted managed to work out that it was an issue with the server not liking empty fragments in the SSL communication. This could be fixed by specifying the `OP_DONT_INSERT_EMPTY_FRAGMENTS` option when making the OpenSSL context.

Currently it seems that in order to be able to scrape a website that has any SSL compatibility issues, you have to subclass `ClientContextFactory` in order to specify the compatibility options, and then subclass HTTPDownloadHandler and tell it to use that context factory.

Could this be made easier? Would it even be a good idea to set some compatibility options by default? Maybe even `SSL.OP_ALL`?