Repository: scrapy-plugins/scrapy-splash

Issue:
Review url being optional for SplashRequest
As part of https://github.com/scrapy-plugins/scrapy-splash/pull/269, the `url` parameter to `SplashRequest` is no longer optional.

@elacuesta noticed that this is a backward-incompatible change. Moreover, the upcoming Scrapy 2.4 will make this change unnecessary (https://github.com/scrapy/scrapy/pull/4835).

We should consider reverting that part of https://github.com/scrapy-plugins/scrapy-splash/pull/269.
Update or set the required Scrapy version
I noticed https://github.com/scrapy-plugins/scrapy-splash/pull/269 did not touch any documentation-like file, but those changes affect to the range of supported Scrapy versions.

I think we need to cover (or update, if already covered) the range of supported Scrapy versions, and probably update `setup.py` file accordingly.


Discussion:
sounds good to me, thanks for the heads up @Gallaecio 
Reverting https://github.com/scrapy-plugins/scrapy-splash/pull/269/commits/4154782b16c6248a5a9b8fe3ed0ebb1cbc600a60 will do just that, so just go for it.
Hi. Is this issue still relevant to the latest version of scrapy-splash? I reverted the changes in https://github.com/scrapy-plugins/scrapy-splash/commit/4154782b16c6248a5a9b8fe3ed0ebb1cbc600a60, ran the tests locally and no failures were noticed.
Well, I think reverting that commit is precisely the goal. I assume you reverted the entire commit, nor just the non-test code, and that keeping the current test code and applying the rest of the commit would result in broken tests.
Yes, I reverted the entire commit including the changes to the tests.
My understanding is that, reverting the commit, [scrapy-splash would fail with scrapy>=2,<2.4](https://github.com/scrapy-plugins/scrapy-splash/pull/269#issuecomment-703901740) if the URL is omitted, because the default URL uses the unsupported `about:` protocol.

The easiest solution I think would be to revert that commit and require scrapy>=2.4.





Patch:
diff --git a/scrapy_splash/request.py b/scrapy_splash/request.py
index 4471054..b3e5819 100644
--- a/scrapy_splash/request.py
+++ b/scrapy_splash/request.py
@@ -31,7 +31,7 @@ class SplashRequest(scrapy.Request):
     It requires SplashMiddleware to work.
     """
     def __init__(self,
-                 url,
+                 url=None,
                  callback=None,
                  method='GET',
                  endpoint='render.html',
@@ -48,6 +48,8 @@ def __init__(self,
                  meta=None,
                  **kwargs):
 
+        if url is None:
+            url = 'about:blank'
         url = to_unicode(url)
 
         meta = copy.deepcopy(meta) or {}
diff --git a/scrapy_splash/utils.py b/scrapy_splash/utils.py
index 3f192f2..03c805c 100644
--- a/scrapy_splash/utils.py
+++ b/scrapy_splash/utils.py
@@ -5,12 +5,7 @@
 import six
 
 from scrapy.http import Headers
-import scrapy
-if scrapy.version_info >= (2, ):
-    from scrapy.utils.python import to_unicode
-else:
-    from scrapy.utils.python import to_native_str as to_unicode
-from scrapy.utils.python import to_bytes
+from scrapy.utils.python import to_unicode, to_bytes
 
 
 def dict_hash(obj, start=''):
diff --git a/setup.py b/setup.py
index 9d1981c..7513d24 100755
--- a/setup.py
+++ b/setup.py
@@ -30,5 +30,5 @@
         'Topic :: Software Development :: Libraries :: Application Frameworks',
         'Topic :: Software Development :: Libraries :: Python Modules',
     ],
-    install_requires=['scrapy', 'six'],
+    install_requires=['scrapy>=2.4', 'six'],
 )


Test patch:
diff --git a/tests/test_fingerprints.py b/tests/test_fingerprints.py
index 22d4b06..09edb19 100644
--- a/tests/test_fingerprints.py
+++ b/tests/test_fingerprints.py
@@ -142,9 +142,9 @@ def requests():
         dict(url=url2, args={'wait': 0.5}),     # 5
         dict(url=url3),                         # 6
         dict(url=url2, method='POST'),          # 7
-        dict(url=url3, args={'wait': 0.5}),     # 8
-        dict(url=url3, args={'wait': 0.5}),     # 9
-        dict(url=url3, args={'wait': 0.7}),     # 10
+        dict(args={'wait': 0.5}),               # 8
+        dict(args={'wait': 0.5}),               # 9
+        dict(args={'wait': 0.7}),               # 10
         dict(url=url4),                         # 11
     ]
     splash_requests = [SplashRequest(**kwargs) for kwargs in request_kwargs]
diff --git a/tests/test_middleware.py b/tests/test_middleware.py
index 76f8c6b..3f53858 100644
--- a/tests/test_middleware.py
+++ b/tests/test_middleware.py
@@ -630,6 +630,21 @@ def test_cache_args():
     assert mw._remote_keys == {}
 
 
+def test_splash_request_no_url():
+    mw = _get_mw()
+    lua_source = "function main(splash) return {result='ok'} end"
+    req1 = SplashRequest(meta={'splash': {
+        'args': {'lua_source': lua_source},
+        'endpoint': 'execute',
+    }})
+    req = mw.process_request(req1, None)
+    assert req.url == 'http://127.0.0.1:8050/execute'
+    assert json.loads(to_unicode(req.body)) == {
+        'url': 'about:blank',
+        'lua_source': lua_source
+    }
+
+
 def test_post_request():
     mw = _get_mw()
     for body in [b'', b'foo=bar']: