Issue 1:
I worked through an example from the documentation on the scrapy shell. One of the commands there did not work however and I got an AlreadyCalledError from a deferred in Twisted.

More specifically, this example failed during the call to request.replace(method="POST"). The method should return a new object according to its doc, but I think the reason Twisted throws the AlreadyCalledError is because it has run through the deferred's callbacks already in the new object. In other words, request.replace() is not returning an independent clone.

Here's a gist containing the exact error message.

Issue 2:
I try to get items from selector, method xpath consider my xpath as a string. I have seen in a book that if i want to iterate over my selector instead of the all page i should use a dot before my xpath, but it seems that doesn't work. This cause i can't get the URL of the article to continue to add items in the items loader from the method parse_article_text. In parse_article_text, i think i don't want to use selector anymore i can use the response to get items from the all page, because i am in the full article page.

Example

I want to get infos from this selector --> article_xpath = "//div[@id='zox-feat-wrap' or @id='zox-home-cont-wrap']//*[contains(@Class,'zox-art-wrap zoxrel')]"

Try to get the title from the selector, this why i use a dot in --> titre_article_xpath = ".//h2[@Class='zox-s-title2' or @Class='zox-s-title1']/text()"

But the true xpath for all the items in the method parse_article begin with //div[@id='zox-feat-wrap' or @id='zox-home-cont-wrap']//*[contains(@Class,'zox-art-wrap zoxrel')]

Ex: the full xpath for the titre_article_xpath mentioned before is //div[@id='zox-feat-wrap' or @id='zox-home-cont-wrap']//*[contains(@Class,'zox-art-wrap zoxrel')]//h2[@Class='zox-s-title2' or @Class='zox-s-title1']/text()

`import scrapy

from ..items import HapiItem

from scrapy.loader import ItemLoader

from itemloaders.processors import MapCompose, Join, Compose

from w3lib.html import replace_escape_chars

class Juno7Spider(scrapy.Spider):

name = 'juno7'

allowed_domains = ['juno7.ht']

url = 'https://www.juno7.ht/{}'
list_category = ('sports/', 'politique/')

def start_requests(self):

for category in Juno7Spider.list_category:

url_category = Juno7Spider.url.format(category)
yield scrapy.Request(url_category, self.parse)

def parse(self, response):

suivant_url_xpath = "//a[normalize-space()='Next ›']/@href"

article_xpath = "//div[@id='zox-feat-wrap' or @id='zox-home-cont-wrap']//*[contains(@class,'zox-art-wrap zoxrel')]"

for article in response.xpath(article_xpath):
yield self.parse_article(selector=article, response=response)

next_page = response.xpath(suivant_url_xpath).get()

if next_page is not None:
yield response.follow(url=next_page, callback=self.parse)

def parse_article(self, selector=None, response=None):
l = ItemLoader(item=HapiItem(), selector=selector)
l.default_input_processor = MapCompose(lambda v: v.split(), replace_escape_chars)
l.default_output_processor = Join()

lien_article_xpath = ".//div[@class='zox-art-title']/a/@href"

titre_article_xpath = ".//h2[@class='zox-s-title2' or @class='zox-s-title1']/text()"

date_article_xpath = ".//div[@class='zox-byline-wrap']/span[@class='zox-byline-date']/text()" # format 8 AVRIL 2021 dd mm yyyy or time AGO jusqu'au 7 jours ex 7 JOURS AGO
image_couverture_article_url_xpath = ".//img[contains(@class,'zox-reg-img')]/@src"

auteur_article_xpath = ".//div[@class='zox-byline-wrap']/span[@class='zox-byline-name']/text()"
category_article_xpath = ".//h3[@class='zox-s-cat']/span/text()"

l.add_xpath('title', titre_article_xpath)
l.add_xpath('url_cover_image', image_couverture_article_url_xpath,
MapCompose(lambda i: response.urljoin(i)))
l.add_value('author', auteur_article_xpath)

l.add_xpath('date', date_article_xpath)
l.add_value('category', category_article_xpath)

# return l.load_item()
url = response.xpath(lien_article_xpath).get()
return response.follow(url=url, callback=self.parse_article_text, cb_kwargs=dict(item=l))

#yield response.follow(url=lien_article_xpath, callback=self.parse_article_text, cb_kwargs=dict(item = l))

def parse_article_text(self, response, item):

texte_article = "//div[contains(@class, 'zox-post-body left zoxrel zox100')]//div[@class='theiaPostSlider_preloadedSlide']/p[position() < (count(//h5[text()='En savoir plus:']/preceding-sibling::*))]/text()"
item.add_xpath('text', texte_article)
item.add_value('url_article', response.url)
return item.load_item()

`

**LOG **

2021-05-05 03:10:42 [scrapy.core.engine] INFO: Spider opened

2021-05-05 03:10:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2021-05-05 03:10:42 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023

2021-05-05 03:10:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/robots.txt> (referer: None)

2021-05-05 03:10:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/> (referer: None)

2021-05-05 03:10:46 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.juno7.ht/jose-mourinho-ap-sou-ban-yon-kleb-italyen-selman-2022/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)

2021-05-05 03:10:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/> (referer: None)

2021-05-05 03:10:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/page/2/> (referer: https://www.juno7.ht/sports/)

2021-05-05 03:10:48 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/page/2/> (referer: https://www.juno7.ht/politique/)

2021-05-05 03:10:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/page/3/> (referer: https://www.juno7.ht/sports/page/2/)

2021-05-05 03:10:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/page/3/> (referer: https://www.juno7.ht/politique/page/2/)

2021-05-05 03:10:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/page/4/> (referer: https://www.juno7.ht/sports/page/3/)

2021-05-05 03:10:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/page/4/> (referer: https://www.juno7.ht/politique/page/3/)

2021-05-05 03:10:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/page/5/> (referer: https://www.juno7.ht/sports/page/4/)

2021-05-05 03:10:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/page/5/> (referer: https://www.juno7.ht/politique/page/4/)

2021-05-05 03:10:53 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/page/6/> (referer: https://www.juno7.ht/sports/page/5/)

2021-05-05 03:10:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/page/6/> (referer: https://www.juno7.ht/politique/page/5/)

2021-05-05 03:10:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/sports/page/7/> (referer: https://www.juno7.ht/sports/page/6/)

2021-05-05 03:10:55 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force

2021-05-05 03:10:55 [scrapy.core.engine] INFO: Closing spider (shutdown)

2021-05-05 03:10:55 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.juno7.ht/politique/page/7/> (referer: https://www.juno7.ht/politique/page/6/)

2021-05-05 03:10:56 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 5387,

'downloader/request_count': 15,

'downloader/request_method_count/GET': 15,

'downloader/response_bytes': 531099,

'downloader/response_count': 15,

'downloader/response_status_count/200': 15,

'dupefilter/filtered': 198,

'elapsed_time_seconds': 14.06602,

'finish_reason': 'shutdown',

'finish_time': datetime.datetime(2021, 5, 5, 7, 10, 56, 112962),

'log_count/DEBUG': 16,

'log_count/INFO': 11,

'request_depth_max': 7,

'response_received_count': 15,

'robotstxt/request_count': 1,

'robotstxt/response_count': 1,

'robotstxt/response_status_count/200': 1,

'scheduler/dequeued': 14,

'scheduler/dequeued/memory': 14,

'scheduler/enqueued': 18,

'scheduler/enqueued/memory': 18,

'start_time': datetime.datetime(2021, 5, 5, 7, 10, 42, 46942)}

2021-05-05 03:10:56 [scrapy.core.engine] INFO: Spider closed (shutdown)