Repository: scrapy/scrapy

Issue:
[Bug] Incorrectly picked URL in `scrapy.http.FormRequest.from_response` when there is a `<base>` tag
## Issue Description

Incorrectly picked URL in `scrapy.http.FormRequest.from_response` when there is a `<base>` tag.
## How to Reproduce the Issue & Version Used

```
[pengyu@GLaDOS tmp]$ python2
Python 2.7.10 (default, Sep  7 2015, 13:51:49) 
[GCC 5.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>> scrapy.__version__
u'1.0.3'
>>> html_body = '''
... <html>
...     <head>
...         <base href="http://b.com/">
...     </head>
...     <body>
...         <form action="test_form">
...         </form>
...     </body>
... </html>
... '''
>>> response = scrapy.http.TextResponse(url='http://a.com/', body=html_body)
>>> request = scrapy.http.FormRequest.from_response(response)
>>> request.url
'http://a.com/test_form'
```
## Expected Result

`request.url` shall be `'http://b.com/test_form'`
## Suggested Fix

The issue can be fixed by fixing a few lines in `scrapy/http/request/form.py`



Discussion:

Resolving patch:
diff --git a/scrapy/http/request/form.py b/scrapy/http/request/form.py
--- a/scrapy/http/request/form.py
+++ b/scrapy/http/request/form.py
@@ -11,6 +11,7 @@
 import six
 from scrapy.http.request import Request
 from scrapy.utils.python import to_bytes, is_listlike
+from scrapy.utils.response import get_base_url
 
 
 class FormRequest(Request):
@@ -44,7 +45,7 @@ def from_response(cls, response, formname=None, formid=None, formnumber=0, formd
 
 def _get_form_url(form, url):
     if url is None:
-        return form.action or form.base_url
+        return urljoin(form.base_url, form.action)
     return urljoin(form.base_url, url)
 
 
@@ -58,7 +59,7 @@ def _urlencode(seq, enc):
 def _get_form(response, formname, formid, formnumber, formxpath):
     """Find the form element """
     text = response.body_as_unicode()
-    root = create_root_node(text, lxml.html.HTMLParser, base_url=response.url)
+    root = create_root_node(text, lxml.html.HTMLParser, base_url=get_base_url(response))
     forms = root.xpath('//form')
     if not forms:
         raise ValueError("No <form> element found in %s" % response)


Test patch:
diff --git a/tests/test_http_request.py b/tests/test_http_request.py
--- a/tests/test_http_request.py
+++ b/tests/test_http_request.py
@@ -801,6 +801,25 @@ def test_from_response_button_novalue(self):
         self.assertEqual(fs[b'test2'], [b'val2'])
         self.assertEqual(fs[b'button1'], [b''])
 
+    def test_html_base_form_action(self):
+        response = _buildresponse(
+            """
+            <html>
+                <head>
+                    <base href="http://b.com/">
+                </head>
+                <body>
+                    <form action="test_form">
+                    </form>
+                </body>
+            </html>
+            """,
+            url='http://a.com/'
+        )
+        req = self.request_class.from_response(response)
+        self.assertEqual(req.url, 'http://b.com/test_form')
+
+
 def _buildresponse(body, **kwargs):
     kwargs.setdefault('body', body)
     kwargs.setdefault('url', 'http://example.com')