Repository: scrapy/scrapy

Issue:
Genspider prepend `http://` without checking it in the `domain`
genspider prepend http:// But when i enter address like https://example.com it becomes http://https://example.com that, when run scrapy crawl throws an error.
What it should do, it should first check the receiving domain than take decision according to the passing domain whether it needs a http:// or nothing.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#5808](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (0261886) into [master](https://codecov.io/gh/scrapy/scrapy/commit/e71eab693264188fe081ebb260baf00cc6b4dc11?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (e71eab6) will **decrease** coverage by `5.47%`.
> The diff coverage is `100.00%`.

> :exclamation: Current head 0261886 differs from pull request most recent head bc1942e. Consider uploading reports for the commit bc1942e to get more accurate results

```diff
@@            Coverage Diff             @@
##           master    #5808      +/-   ##
==========================================
- Coverage   88.91%   83.44%   -5.47%     
==========================================
  Files         162      162              
  Lines       10988    10989       +1     
  Branches     1797     1796       -1     
==========================================
- Hits         9770     9170     -600     
- Misses        938     1551     +613     
+ Partials      280      268      -12     
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/commands/genspider.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2dlbnNwaWRlci5weQ==) | `87.27% <100.00%> (-0.12%)` | :arrow_down: |
| [scrapy/utils/template.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3RlbXBsYXRlLnB5) | `100.00% <100.00%> (ø)` | |
| [scrapy/core/http2/stream.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvc3RyZWFtLnB5) | `0.00% <0.00%> (-91.91%)` | :arrow_down: |
| [scrapy/core/downloader/handlers/http2.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvZG93bmxvYWRlci9oYW5kbGVycy9odHRwMi5weQ==) | `14.08% <0.00%> (-85.92%)` | :arrow_down: |
| [scrapy/core/http2/agent.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvYWdlbnQucHk=) | `13.25% <0.00%> (-83.14%)` | :arrow_down: |
| [scrapy/core/http2/protocol.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvcHJvdG9jb2wucHk=) | `3.51% <0.00%> (-79.90%)` | :arrow_down: |
| [scrapy/pipelines/images.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3BpcGVsaW5lcy9pbWFnZXMucHk=) | `26.47% <0.00%> (-70.59%)` | :arrow_down: |
| [scrapy/utils/test.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3Rlc3QucHk=) | `51.35% <0.00%> (-14.87%)` | :arrow_down: |
| [scrapy/robotstxt.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3JvYm90c3R4dC5weQ==) | `75.30% <0.00%> (-13.59%)` | :arrow_down: |
| [scrapy/core/downloader/contextfactory.py](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvZG93bmxvYWRlci9jb250ZXh0ZmFjdG9yeS5weQ==) | `76.78% <0.00%> (-10.72%)` | :arrow_down: |
| ... and [9 more](https://codecov.io/gh/scrapy/scrapy/pull/5808?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | |

alexpdev: It seems to be failing while testing a download handlers:

https://github.com/scrapy/scrapy/actions/runs/4028043270/jobs/6924481148#step:4:97
Gallaecio: Regarding CI, I‘ve just started a re-run of the failing job, as I suspect it was a run-time issue, not an issue with your changes.


wRAR: Can we just change the templates to always expect the URL and change `render_templatefile` to construct an URL if only a domain was passed?
alexpdev: > Can we just change the templates to always expect the URL and change `render_templatefile` to construct an URL if only a domain was passed?

Probably, Should the URL scheme default to 'http' or 'https' in cases where it is just a domain?
wRAR: > Should the URL scheme default to 'http' or 'https' in cases where it is just a domain?

I think https is fine.

There may be some backwards compatibility considerations both because of this question and because of support for 3rd-party templates, but I don't know if we should consider them. 

Patch:
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index 54fd5d663..106045fc0 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -238,9 +238,6 @@ genspider
 
 Create a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ``<name>`` parameter is set as the spider's ``name``, while ``<domain or URL>`` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes.
 
-.. note:: Even if an HTTPS URL is specified, the protocol used in
-          ``start_urls`` is always HTTP. This is a known issue: :issue:`3553`.
-
 Usage example::
 
     $ scrapy genspider -l
diff --git a/scrapy/commands/genspider.py b/scrapy/commands/genspider.py
index c1565a138..68cbe8ff6 100644
--- a/scrapy/commands/genspider.py
+++ b/scrapy/commands/genspider.py
@@ -31,6 +31,14 @@ def extract_domain(url):
     return o.netloc
 
 
+def verify_url_scheme(url):
+    """Check url for scheme and insert https if none found."""
+    parsed = urlparse(url)
+    if parsed.scheme == "" and parsed.netloc == "":
+        parsed = urlparse("//" + url)._replace(scheme="https")
+    return parsed.geturl()
+
+
 class Command(ScrapyCommand):
     requires_project = False
     default_settings = {"LOG_ENABLED": False}
@@ -91,7 +99,7 @@ class Command(ScrapyCommand):
             raise UsageError()
 
         name, url = args[0:2]
-        domain = extract_domain(url)
+        url = verify_url_scheme(url)
         module = sanitize_module_name(name)
 
         if self.settings.get("BOT_NAME") == module:
@@ -103,18 +111,20 @@ class Command(ScrapyCommand):
 
         template_file = self._find_template(opts.template)
         if template_file:
-            self._genspider(module, name, domain, opts.template, template_file)
+            self._genspider(module, name, url, opts.template, template_file)
             if opts.edit:
                 self.exitcode = os.system(f'scrapy edit "{name}"')
 
-    def _genspider(self, module, name, domain, template_name, template_file):
+    def _genspider(self, module, name, url, template_name, template_file):
         """Generate the spider module, based on the given template"""
         capitalized_module = "".join(s.capitalize() for s in module.split("_"))
+        domain = extract_domain(url)
         tvars = {
             "project_name": self.settings.get("BOT_NAME"),
             "ProjectName": string_camelcase(self.settings.get("BOT_NAME")),
             "module": module,
             "name": name,
+            "url": url,
             "domain": domain,
             "classname": f"{capitalized_module}Spider",
         }
diff --git a/scrapy/templates/spiders/basic.tmpl b/scrapy/templates/spiders/basic.tmpl
index d3ba19553..20e777271 100644
--- a/scrapy/templates/spiders/basic.tmpl
+++ b/scrapy/templates/spiders/basic.tmpl
@@ -4,7 +4,7 @@ import scrapy
 class $classname(scrapy.Spider):
     name = "$name"
     allowed_domains = ["$domain"]
-    start_urls = ["http://$domain/"]
+    start_urls = ["$url"]
 
     def parse(self, response):
         pass
diff --git a/scrapy/templates/spiders/crawl.tmpl b/scrapy/templates/spiders/crawl.tmpl
index 2e467e632..36d05e43a 100644
--- a/scrapy/templates/spiders/crawl.tmpl
+++ b/scrapy/templates/spiders/crawl.tmpl
@@ -6,7 +6,7 @@ from scrapy.spiders import CrawlSpider, Rule
 class $classname(CrawlSpider):
     name = "$name"
     allowed_domains = ["$domain"]
-    start_urls = ["http://$domain/"]
+    start_urls = ["$url"]
 
     rules = (Rule(LinkExtractor(allow=r"Items/"), callback="parse_item", follow=True),)
 
diff --git a/scrapy/templates/spiders/csvfeed.tmpl b/scrapy/templates/spiders/csvfeed.tmpl
index ce9c1dd20..fe96878dc 100644
--- a/scrapy/templates/spiders/csvfeed.tmpl
+++ b/scrapy/templates/spiders/csvfeed.tmpl
@@ -4,7 +4,7 @@ from scrapy.spiders import CSVFeedSpider
 class $classname(CSVFeedSpider):
     name = "$name"
     allowed_domains = ["$domain"]
-    start_urls = ["http://$domain/feed.csv"]
+    start_urls = ["$url"]
     #headers = ["id", "name", "description", "image_link"]
     #delimiter = "\t"
 
diff --git a/scrapy/templates/spiders/xmlfeed.tmpl b/scrapy/templates/spiders/xmlfeed.tmpl
index 6b50e4cf4..ac62d78d1 100644
--- a/scrapy/templates/spiders/xmlfeed.tmpl
+++ b/scrapy/templates/spiders/xmlfeed.tmpl
@@ -4,7 +4,7 @@ from scrapy.spiders import XMLFeedSpider
 class $classname(XMLFeedSpider):
     name = "$name"
     allowed_domains = ["$domain"]
-    start_urls = ["http://$domain/feed.xml"]
+    start_urls = ["$url"]
     iterator = "iternodes"  # you can change this; see the docs
     itertag = "item"  # change it accordingly
 


Test patch:
diff --git a/tests/test_commands.py b/tests/test_commands.py
index 00ddcdd3e..014f50e92 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -541,7 +541,7 @@ class GenspiderCommandTest(CommandTest):
             ).group(1),
         )
         self.assertEqual(
-            f"http://{domain}/",
+            f"https://{domain}",
             self.find_in_file(
                 Path(self.proj_mod_path, "spiders", "test_name.py"),
                 r"start_urls\s*=\s*\[['\"](.+)['\"]\]",
@@ -549,13 +549,64 @@ class GenspiderCommandTest(CommandTest):
         )
 
     def test_url_schema(self):
-        self.test_url("http://test.com", "test.com")
+        self.test_url("https://test.com", "test.com")
 
-    def test_url_path(self):
-        self.test_url("test.com/some/other/page", "test.com")
+    def test_template_start_urls(
+        self, url="test.com", expected="https://test.com", template="basic"
+    ):
+        self.assertEqual(
+            0, self.call("genspider", "-t", template, "--force", "test_name", url)
+        )
+        self.assertEqual(
+            expected,
+            self.find_in_file(
+                Path(self.proj_mod_path, "spiders", "test_name.py"),
+                r"start_urls\s*=\s*\[['\"](.+)['\"]\]",
+            ).group(1),
+        )
+
+    def test_genspider_basic_start_urls(self):
+        self.test_template_start_urls("https://test.com", "https://test.com", "basic")
+        self.test_template_start_urls("http://test.com", "http://test.com", "basic")
+        self.test_template_start_urls(
+            "http://test.com/other/path", "http://test.com/other/path", "basic"
+        )
+        self.test_template_start_urls(
+            "test.com/other/path", "https://test.com/other/path", "basic"
+        )
 
-    def test_url_schema_path(self):
-        self.test_url("https://test.com/some/other/page", "test.com")
+    def test_genspider_crawl_start_urls(self):
+        self.test_template_start_urls("https://test.com", "https://test.com", "crawl")
+        self.test_template_start_urls("http://test.com", "http://test.com", "crawl")
+        self.test_template_start_urls(
+            "http://test.com/other/path", "http://test.com/other/path", "crawl"
+        )
+        self.test_template_start_urls(
+            "test.com/other/path", "https://test.com/other/path", "crawl"
+        )
+        self.test_template_start_urls("test.com", "https://test.com", "crawl")
+
+    def test_genspider_xmlfeed_start_urls(self):
+        self.test_template_start_urls(
+            "https://test.com/feed.xml", "https://test.com/feed.xml", "xmlfeed"
+        )
+        self.test_template_start_urls(
+            "http://test.com/feed.xml", "http://test.com/feed.xml", "xmlfeed"
+        )
+        self.test_template_start_urls(
+            "test.com/feed.xml", "https://test.com/feed.xml", "xmlfeed"
+        )
+
+    def test_genspider_csvfeed_start_urls(self):
+        self.test_template_start_urls(
+            "https://test.com/feed.csv", "https://test.com/feed.csv", "csvfeed"
+        )
+        self.test_template_start_urls(
+            "http://test.com/feed.xml", "http://test.com/feed.xml", "csvfeed"
+        )
+        self.test_template_start_urls(
+            "test.com/feed.csv", "https://test.com/feed.csv", "csvfeed"
+        )
 
 
 class GenspiderStandaloneCommandTest(ProjectTest):