Issue 1:
Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/scrapy/core/downloader/middleware.py", line 43, in process_request

defer.returnValue((yield download_func(request=request,spider=spider)))

ValueError: invalid hostname: advisors-dev-mra-2333_profile_update.swarm.cxawsnprd.massmutual.com

ERROR:scrapy.core.scraper:Error downloading <GET https://advisors-dev-mra-2333_profile_update.swarm.cxawsnprd.massmutual.com/shealyn-mcgaffey>

Scrapy : 1.3.0

lxml : 4.2.5.0

libxml2 : 2.9.8

cssselect : 1.0.3

parsel : 1.5.0

w3lib : 1.19.0

Twisted : 18.7.0

Python : 2.7.15rc1 (default, Apr 15 2018, 21:51:34) - [GCC 7.3.0]

pyOpenSSL : 18.0.0 (OpenSSL 1.1.0i 14 Aug 2018)

Issue 2:
I have url with invalid hostname - it does not match IDNA standards. Scrapy fails with that.

scrapy fetch "https://mediaworld_it_api2.frosmo.com/?method=products&products=[%22747190%22]"

2018-07-06 11:53:09 [scrapy.core.scraper] ERROR: Error downloading <GET https://mediaworld_it_api2.frosmo.com/?method=products&products=[%22747190%22]>
Traceback (most recent call last):
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/pawel/scrapy/scrapy/core/downloader/middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
File "/home/pawel//scrapy/scrapy/utils/defer.py", line 45, in mustbe_deferred
result = f(*args, **kw)
File "/home/pawel/scrapy/scrapy/core/downloader/handlers/__init__.py", line 65, in download_request
return handler.download_request(request, spider)
File "/home/pawel/scrapy/scrapy/core/downloader/handlers/http11.py", line 67, in download_request
return agent.download_request(request)
File "/home/pawelscrapy/scrapy/core/downloader/handlers/http11.py", line 331, in download_request
method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py", line 1649, in request
endpoint = self._getEndpoint(parsedURI)
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py", line 1633, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/web/client.py", line 1510, in endpointForURI
uri.port)
File "/home/pawel/scrapy/scrapy/core/downloader/contextfactory.py", line 59, in creatorForNetloc
return ScrapyClientTLSOptions(hostname.decode("ascii"), self.getContext())
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1152, in __init__
self._hostnameBytes = _idnaBytes(hostname)
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/twisted/internet/_idna.py", line 30, in _idnaBytes
return idna.encode(text)
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.py", line 355, in encode
result.append(alabel(label))
File "/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.py", line 265, in alabel
raise IDNAError('The label {0} is not a valid A-label'.format(label))
IDNAError: The label mediaworld_it_api2 is not a valid A-label

IDNA error is legitmate. This url https://mediaworld_it_api2.frosmo.com/?method=products&products=[%22747190%22] is not valid according to IDNA standard.

In [1]: x = "https://mediaworld_it_api2.frosmo.com/?method=products&products=[%22747190%22]"

In [2]: import idna

In [3]: idna.encode(x)
---------------------------------------------------------------------------
IDNAError Traceback (most recent call last)
<ipython-input-3-c97070e17b57> in <module>()
----> 1 idna.encode(x)

/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.pyc in encode(s, strict, uts46, std3_rules, transitional)
353 trailing_dot = True
354 for label in labels:
--> 355 result.append(alabel(label))
356 if trailing_dot:
357 result.append(b'')

/home/pawel/.virtualenvs/scrapy/local/lib/python2.7/site-packages/idna/core.pyc in alabel(label)
263 ulabel(label)
264 except IDNAError:
--> 265 raise IDNAError('The label {0} is not a valid A-label'.format(label))
266 if not valid_label_length(label):
267 raise IDNAError('Label too long')

IDNAError: The label https://mediaworld_it_api2 is not a valid A-label

How should scrapy handle this url? Should we download it regardless of validity?