I was unable to set a user agent otherwise and to get the page headers (required for pagination)