Issue 1:
Hi all,

My name is Martin, a master candidate in information system at Carnegie Mellon. I have worked as an analyst and product manager before, but I would like to become more involved in the development space and contribute more to open source projects.

I have personally used scrapy a lot, using it both for personal projects as well as for work. I would be extremely excited to contribute to this project through GoSC.

I find the request fingerprints project very interesting, as I have personally faced issues with this before. For example, when I scraped Instagram, the website uses a "session_id" to check on the frequency of requests. The default behavior works in this case, but if I want to see how different users sees the same URL (for example, the recommendation page), it would be hard with the current framework.

I checked the source code and found the fingerprint is implemented here.

_fingerprint_cache = weakref.WeakKeyDictionary()
def request_fingerprint(request, include_headers=None):
if include_headers:
include_headers = tuple(to_bytes(h.lower())
for h in sorted(include_headers))
cache = _fingerprint_cache.setdefault(request, {})
if include_headers not in cache:
fp = ()
fp.update(to_bytes(request.method))
fp.update(to_bytes(canonicalize_url(request.url)))
fp.update(request.body or b'')
if include_headers:
for hdr in include_headers:
if hdr in request.headers:
fp.update(hdr)
for v in request.headers.getlist(hdr):
fp.update(v)
cache[include_headers] = fp.hexdigest()
return cache[include_headers]

The function is used in the dupefilter class, but the optional parameter "include_header" is never used anywhere in the project. The documentation (see below) says that you can customize this by overwriting the request_fingerprint function. However, this will be out of scope for most users who did the pip install, and I think users should be provided more convinient ways to adjust this setting.

The default (RFPDupeFilter) filters based on request fingerprint using the scrapy.utils.request.request_fingerprint function. In order to change the way duplicates are checked you could subclass RFPDupeFilter and override its request_fingerprint method. This method should accept scrapy Request object and return its fingerprint (a string).

On the idea page, it says

Scrapy uses a Request fingerprinting scheme for de-duplicating requests and for caching. Currently, the fingerprinting algorithm cannot be modified by Scrapy users.

No use cases were mentioned here. The use case I can think of is to provide user simple ways to treat certain headers as part of the fingerprint (for example, language settings or session_id for some sites). Under the current implementation, even if the user chooses to enable the "include_headers" parameter, he/she cannot specify the part of headers to be included, meaning things like rotating User-Agent will cause the program to crawl the same page multiple times.

Those are just my two cents on this issue. If anyone can provide more use cases on this, it can really help me think this through.

@Gallaecio If you can provide some comment on this it would be great :)

Thanks in advance,

Martin

Issue 2:
import logging

import coloredlogs

logging.captureWarnings(True)

LOGGER = logging.getLogger(__name__)

coloredlogs.install(level='INFO')

Initialization is done this way and translates into a duplication of logs in the output. Any solution to keep only the colored logs?