Repository: scrapy/scrapy

Issue:
Genspider prepend `http://` without checking it in the `domain`
genspider prepend http:// But when i enter address like https://example.com it becomes http://https://example.com that, when run scrapy crawl throws an error.
What it should do, it should first check the receiving domain than take decision according to the passing domain whether it needs a http:// or nothing.
Fix [#3553] Add_http_if_no_scheme in domain in genspider
Fixed [#3553] 
before passing the domain, use built-in function from utils.url.add_http_if_no_scheme to add the http:// before the domain.
Updated template files by removing hard coded values.


Discussion:
Hello everybody! I would like to contribute to Scrapy, specifically with this issue and I already fork the repo and made the changes but before issuing a pull request I would like to discuss it and reach to an agreement.

I made the changes over 5 files:
- **scrapy/commands/genspider.py**. Where `urllib.parse.urlparse` is being used on the domain parameter and prepend `http` scheme in the case is not there. But, this is a very tricky situation because of the command syntax, which asks for a domain and not an URL, so we're just putting a very big bet that it should be HTTP but what about if we ask for a URL instead of a domain, then the contradiction is solved right away since we can extract the domain without making any assumption. Don't you think fellas?
- **Each template file**. In order to receive domain and URL values.
I checked the test just for tests/test_commands.py, I didn't add test because I don't know how to make a test for this particular scenario where two attributes are set based on user input on a class created through a template. If someone has an idea how it could be made please I'm all ears

Lastly and maybe more important, I don't know how to try the code it's my first time and I don't have any idea how to execute it. So any help or guide would be really appreciated.

Thank you in advance and happy coding!!!! =D
@ambarmendez I would suggest you open a pull request already, discussions over actual code are usually easier.

Also, make sure you check out previous pull requests (#3554, #3558). Feedback there, specially @kmike’s, could answer some of your points.
Thank you @Gallaecio! Let's see how it goes =D
Now URLs are supported as input: https://github.com/scrapy/scrapy/pull/4439

However, we still simply extract the domain from them. Which means that the input protocol is not respected for `start_urls`, and instead it is always forced to be HTTP (as opposed to, for example, HTTPS).
https://github.com/scrapy/scrapy/pull/3558 seems to aim to address the `start_urls` issue. The pull request needs some work, but if not resumed, maybe it can be taken as inspiration for alternative implementations.
Hi @Gallaecio, can I be assigned to work on this issue?
@msenior85 No need. Feel free to open a pull request and include `Fixes #3553` in its description so that it shows up in this thread.
# [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/3558?src=pr&el=h1) Report
> Merging [#3558](https://codecov.io/gh/scrapy/scrapy/pull/3558?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/scrapy/commit/094dde6fdb1b03351888e437828af5da03f46352?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `76.92%`.


```diff
@@            Coverage Diff             @@
##           master    #3558      +/-   ##
==========================================
- Coverage   84.48%   84.47%   -0.02%     
==========================================
  Files         167      167              
  Lines        9405     9416      +11     
  Branches     1397     1399       +2     
==========================================
+ Hits         7946     7954       +8     
- Misses       1201     1202       +1     
- Partials      258      260       +2
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/3558?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [scrapy/commands/genspider.py](https://codecov.io/gh/scrapy/scrapy/pull/3558/diff?src=pr&el=tree#diff-c2NyYXB5L2NvbW1hbmRzL2dlbnNwaWRlci5weQ==) | `82.47% <76.92%> (-1.25%)` | :arrow_down: |


Resolving patch:
diff --git a/scrapy/commands/genspider.py b/scrapy/commands/genspider.py
--- a/scrapy/commands/genspider.py
+++ b/scrapy/commands/genspider.py
@@ -31,6 +31,14 @@ def extract_domain(url):
     return o.netloc
 
 
+def verify_url_scheme(url):
+    """Check url for scheme and insert https if none found."""
+    parsed = urlparse(url)
+    if parsed.scheme == "" and parsed.netloc == "":
+        parsed = urlparse("//" + url)._replace(scheme="https")
+    return parsed.geturl()
+
+
 class Command(ScrapyCommand):
     requires_project = False
     default_settings = {"LOG_ENABLED": False}
@@ -91,7 +99,7 @@ def run(self, args, opts):
             raise UsageError()
 
         name, url = args[0:2]
-        domain = extract_domain(url)
+        url = verify_url_scheme(url)
         module = sanitize_module_name(name)
 
         if self.settings.get("BOT_NAME") == module:
@@ -103,18 +111,20 @@ def run(self, args, opts):
 
         template_file = self._find_template(opts.template)
         if template_file:
-            self._genspider(module, name, domain, opts.template, template_file)
+            self._genspider(module, name, url, opts.template, template_file)
             if opts.edit:
                 self.exitcode = os.system(f'scrapy edit "{name}"')
 
-    def _genspider(self, module, name, domain, template_name, template_file):
+    def _genspider(self, module, name, url, template_name, template_file):
         """Generate the spider module, based on the given template"""
         capitalized_module = "".join(s.capitalize() for s in module.split("_"))
+        domain = extract_domain(url)
         tvars = {
             "project_name": self.settings.get("BOT_NAME"),
             "ProjectName": string_camelcase(self.settings.get("BOT_NAME")),
             "module": module,
             "name": name,
+            "url": url,
             "domain": domain,
             "classname": f"{capitalized_module}Spider",
         }


Test patch:
diff --git a/tests/test_commands.py b/tests/test_commands.py
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -541,7 +541,7 @@ def test_url(self, url="test.com", domain="test.com"):
             ).group(1),
         )
         self.assertEqual(
-            f"http://{domain}/",
+            f"https://{domain}",
             self.find_in_file(
                 Path(self.proj_mod_path, "spiders", "test_name.py"),
                 r"start_urls\s*=\s*\[['\"](.+)['\"]\]",
@@ -549,13 +549,64 @@ def test_url(self, url="test.com", domain="test.com"):
         )
 
     def test_url_schema(self):
-        self.test_url("http://test.com", "test.com")
+        self.test_url("https://test.com", "test.com")
 
-    def test_url_path(self):
-        self.test_url("test.com/some/other/page", "test.com")
+    def test_template_start_urls(
+        self, url="test.com", expected="https://test.com", template="basic"
+    ):
+        self.assertEqual(
+            0, self.call("genspider", "-t", template, "--force", "test_name", url)
+        )
+        self.assertEqual(
+            expected,
+            self.find_in_file(
+                Path(self.proj_mod_path, "spiders", "test_name.py"),
+                r"start_urls\s*=\s*\[['\"](.+)['\"]\]",
+            ).group(1),
+        )
+
+    def test_genspider_basic_start_urls(self):
+        self.test_template_start_urls("https://test.com", "https://test.com", "basic")
+        self.test_template_start_urls("http://test.com", "http://test.com", "basic")
+        self.test_template_start_urls(
+            "http://test.com/other/path", "http://test.com/other/path", "basic"
+        )
+        self.test_template_start_urls(
+            "test.com/other/path", "https://test.com/other/path", "basic"
+        )
 
-    def test_url_schema_path(self):
-        self.test_url("https://test.com/some/other/page", "test.com")
+    def test_genspider_crawl_start_urls(self):
+        self.test_template_start_urls("https://test.com", "https://test.com", "crawl")
+        self.test_template_start_urls("http://test.com", "http://test.com", "crawl")
+        self.test_template_start_urls(
+            "http://test.com/other/path", "http://test.com/other/path", "crawl"
+        )
+        self.test_template_start_urls(
+            "test.com/other/path", "https://test.com/other/path", "crawl"
+        )
+        self.test_template_start_urls("test.com", "https://test.com", "crawl")
+
+    def test_genspider_xmlfeed_start_urls(self):
+        self.test_template_start_urls(
+            "https://test.com/feed.xml", "https://test.com/feed.xml", "xmlfeed"
+        )
+        self.test_template_start_urls(
+            "http://test.com/feed.xml", "http://test.com/feed.xml", "xmlfeed"
+        )
+        self.test_template_start_urls(
+            "test.com/feed.xml", "https://test.com/feed.xml", "xmlfeed"
+        )
+
+    def test_genspider_csvfeed_start_urls(self):
+        self.test_template_start_urls(
+            "https://test.com/feed.csv", "https://test.com/feed.csv", "csvfeed"
+        )
+        self.test_template_start_urls(
+            "http://test.com/feed.xml", "http://test.com/feed.xml", "csvfeed"
+        )
+        self.test_template_start_urls(
+            "test.com/feed.csv", "https://test.com/feed.csv", "csvfeed"
+        )
 
 
 class GenspiderStandaloneCommandTest(ProjectTest):