dakrone/clj-http — pull request #27 by jli (2011-12-13 06:27:52+00:00)
Title: only use :server-port for non-standard ports

When :server-port is provided (by client/parse-url), it's included in the url built by core/request:

(let [http-url (str scheme "://" server-name
(when server-port (str ":" server-port))
uri
(when query-string (str "?" query-string)))
...] ...)

However, for standard ports (80 for http, 443 for https), this causes problems on some servers.

I get a plain 404 from Flickr:
https://secure.flickr.com/photos/marcjohns/2361135683/in/photostream/
clj-http.client> (head "https://secure.flickr.com/photos/marcjohns/2341099354/" {:throw-exceptions false })
{:status 404, :headers {"server" "YTS/1.19.4", "content-language" "en", "via" "HTTP/1.1 r17.ycpi.mud.yahoo.net (YahooTrafficServer/1.20.9 [c s f ])", "content-type" "text/html", "date" "Tue, 13 Dec 2011 06:14:57 GMT", "cache-control" "no-store", "content-length" "240", "connection" "close", "age" "0"}, :body nil}

And from this other site, I get stuck in a 301 loop:

clj-http.client> (head "http://hipsteripsum.me/" {:throw-exceptions false :debug true})
Request:
{:scheme "http",
:http-url "http://hipsteripsum.me:80/",
:request-method :head,
:query-string nil,
:uri "/",
:server-name "hipsteripsum.me",
:headers {"Accept-Encoding" "gzip, deflate"},
:debug true,
:server-port 80,
:throw-exceptions false,
:body "... 0 bytes ...",
:user-info nil}
HttpRequest:
{:requestLine #,
:protocolVersion #,
:params #,
:method "HEAD",
:class org.apache.http.client.methods.HttpHead,
:allHeaders [#, #],
:aborted false,
:URI #}
Request:
{:scheme "http",
:http-url "http://hipsteripsum.me:80/",
:request-method :head,
...

Investigating with a packet dump, it looks like this server is sending 301s because in the request, I'm sending "Host: hipsteripsum.me:80". Reproducible with nc:
$ echo -e 'HEAD / HTTP/1.1\nHost: hipsteripsum.me\n' | nc hipsteripsum.me 80
HTTP/1.1 200 OK
Date: Tue, 13 Dec 2011 06:23:23 GMT
Server: Apache
Vary: Cookie,Accept-Encoding
X-Pingback: http://hipsteripsum.me/xmlrpc.php
Last-Modified: Tue, 13 Dec 2011 06:01:40 GMT
Content-Type: text/html; charset=UTF-8

Now, with ":80":
$ echo -e 'HEAD / HTTP/1.1\nHost: hipsteripsum.me:80\n' | nc hipsteripsum.me 80
HTTP/1.1 301 Moved Permanently
Date: Tue, 13 Dec 2011 06:23:37 GMT
Server: Apache
X-Pingback: http://hipsteripsum.me/xmlrpc.php
Location: http://hipsteripsum.me/
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8

I originally changed core/request to not include the port if:
(or (and (= scheme "http") (= port 80))
(and (= scheme "https") (= port 443)))
But I think the current change is better, though it does change semantics and I had to modify a test.