Issue 1:
Description

I send some custom_settings values through api. While these values are successfully received in kwargs, they are not applied in crawler settings.

I wrote two different codes for the same thing, but I didn't succeed, and maybe scrapy has a bug in this part or I made a mistake.

First code:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.loader import ItemLoader
from kavoush.lxmlhtml import LxmlLinkExtractor as LinkExtractor
from kavoush.items import PageLevelItem

class PageSpider(CrawlSpider):
name = 'github'

def __init__(self, *args, **kwargs):
self.start_urls = kwargs.get('host_name')
self.allowed_domains = [self.start_urls]
self.custom_settings = {
'CONCURRENT_REQUESTS': int(kwargs.get('CONCURRENT_REQUESTS')),
'ROBOTSTXT_OBEY': kwargs.get('ROBOTSTXT_OBEY') == 'True',
}
self.logger.info(f'CONCURRENT_REQUESTS? {self.custom_settings}')

self.rules = (
Rule(LinkExtractor(allow=(self.start_urls),deny=('\.webp'),unique=True),
callback='parse',
follow=True),
)
super(PageSpider, self).__init__(*args, **kwargs)

def parse(self,response):
loader = ItemLoader(item=PageLevelItem(), response=response)
loader.add_xpath('page_source_html_lang', "//html/@lang")
yield loader.load_item()

def errback_domain(self, failure):
self.logger.error(repr(failure))

And according to the photo, the settings I want via api are not available in Overridden settings, but in the log, the correct values are sent and received successfully.

The photo below is the request that I sent through api and with the help of scrapyd.

The second code has the same problem, actually I expected one of the codes (the first code or the second code) to work, but none of them applies the settings.

Second code:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.loader import ItemLoader
from kavoush.lxmlhtml import LxmlLinkExtractor as LinkExtractor
from kavoush.items import PageLevelItem
from scrapy.settings import Settings

class PageSpider(CrawlSpider):
name = 'github1'

def __init__(self, *args, **kwargs):
self.start_urls = kwargs.get('host_name')
self.allowed_domains = [self.start_urls]
self.custom_settings = {
'CONCURRENT_REQUESTS': int(kwargs.get('CONCURRENT_REQUESTS')),
'ROBOTSTXT_OBEY': kwargs.get('ROBOTSTXT_OBEY') == 'True',
}
self.logger.info(f'CONCURRENT_REQUESTS? {self.custom_settings}')

for key, value in kwargs.items():
if key in self.custom_settings:
self.custom_settings[key] = value

settings = Settings()
settings.setdict(self.custom_settings, priority='spider')
self.settings = settings

self.rules = (
Rule(LinkExtractor(allow=(self.start_urls),deny=('\.webp'),unique=True),
callback='parse',
follow=True),
)
super(PageSpider, self).__init__(*args, **kwargs)

def parse(self,response):
loader = ItemLoader(item=PageLevelItem(), response=response)
loader.add_xpath('page_source_html_lang', "//html/@lang")
yield loader.load_item()

def errback_domain(self, failure):
self.logger.error(repr(failure))

Versions

Scrapy : 2.7.1
lxml : 4.9.2.0
libxml2 : 2.9.14
cssselect : 1.2.0
parsel : 1.7.0
w3lib : 2.1.1
Twisted : 22.10.0
Python : 3.10.9
pyOpenSSL : 22.1.0 (OpenSSL 3.0.7 1 Nov 2022)
cryptography : 38.0.1
Platform : Linux-5.15.0-50-generic-x86_64-with-glibc2.35

Additional context

I already asked my question in stackoverflow and said maybe I don't know, but the answer in stackoverflow was similar to my first code, but it doesn't work in scrapy.

In short, I want to send some custom_settings via api, but I didn't succeed.

Thank you friends.

Issue 2:
This issue might be related to #1305

I noticed that settings are frozen in https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L57

However, in a given project I had a requirement to change some settings based on some spider arguments. An alternative would be to write this spider as a base class and extend it from specific spiders setting the proper settings.

However, I think it would make sense to only freeze settings after the spider and other components were initialized. Or, provide some other entry point to configure settings based on arguments.

The other option is to use -s arguments, but in my case I was changing the FEED_EXPORT_FIELDS setting (https://docs.scrapy.org/en/latest/topics/feed-exports.html#std:setting-FEED_EXPORT_FIELDS).

Any thoughts here?