Repository: scrapy/scrapy

Issue:
Scheduler initializes disk queue even if `JOBDIR` is empty string
# Description

By default, `JOBDIR` is the empty string per documentation. This makes `SpiderState` extension to be disabled by default and `RFPDupeFilter` to not save fingerprints to file `requests.seen`. However, request scheduler still creates disk queue. The leads to inconsistences: pending requests will be saved to the disk, but their fingerprints and spider state will not.

Actually, there is no default value for `JOBDIR` at all, and the documentation is wrong. This setting is never set. And by design of `BaseSettings` unset settings are `None`.

# Steps to Reproduce

1. Create new project.
2. Create simple spider which yields one requests and then exists.
3. Open terminal and cd to the root of the project.
4. Start it with `scrapy crawl spider_name -s JOBDIR=''`.
5. Wait till spider closure.

Sample spider code:
```python
class TestSpider(BaseSpider):
    name = '_test'

    def start_requests(self, /):
        return Request('data:,', self.check)

    def check(self, _, /): pass
```

**Expected behavior:** Root directory of the project does not contain directory `requests.queue`.

**Actual behavior:** Root directory of the project contains directory `requests.queue`.

**Reproduces how often:** always.

# Versions

```
Scrapy       : 2.11.0
lxml         : 4.9.3.0
libxml2      : 2.10.3
cssselect    : 1.2.0
parsel       : 1.8.1
w3lib        : 2.1.2
Twisted      : 22.10.0
Python       : 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)]
pyOpenSSL    : 23.2.0 (OpenSSL 3.1.3 19 Sep 2023)
cryptography : 41.0.4
Platform     : Windows-10-10.0.19044-SP0
```

# The fix

The reason of this behavior is very simple. `SpiderState` and `RFPDupeFilter` performs truth checks for `JOBDIR`, but request scheduler checks whether it is not `None`.

I suggest the next steps to eliminate the issue:

1. In documentation change `JOBDIR` default value from `''` to `None` because it is the actual default value.
2. In scheduler also perform truth check. [Here](https://github.com/scrapy/scrapy/blob/9b06f6b316b2759b35b1ec39ef93ccb563458c9c/scrapy/core/scheduler.py#L355) change `if jobdir is not None:` to `if jobdir:`.

The next suggestions are optional, but can help eliminate any future issues:

1. Change function [`jobdir`](https://github.com/scrapy/scrapy/blob/9b06f6b316b2759b35b1ec39ef93ccb563458c9c/scrapy/utils/job.py#L7) to return `None` if `JOBDIR` is the empty string.
   - This change allows to check job directory presence by truth and `is not None`.
2. Explicitly set `JOBDIR` in default settings to `None`.
   - This change enables displaying custom `JOBDIR` in the `Overridden settings` log.



Discussion:

Resolving patch:
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -352,7 +352,7 @@ def _dq(self):
 
     def _dqdir(self, jobdir: Optional[str]) -> Optional[str]:
         """Return a folder name to keep disk queue state at"""
-        if jobdir is not None:
+        if jobdir:
             dqdir = Path(jobdir, "requests.queue")
             if not dqdir.exists():
                 dqdir.mkdir(parents=True)
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -206,6 +206,8 @@
 ITEM_PIPELINES = {}
 ITEM_PIPELINES_BASE = {}
 
+JOBDIR = None
+
 LOG_ENABLED = True
 LOG_ENCODING = "utf-8"
 LOG_FORMATTER = "scrapy.logformatter.LogFormatter"
diff --git a/scrapy/utils/job.py b/scrapy/utils/job.py
--- a/scrapy/utils/job.py
+++ b/scrapy/utils/job.py
@@ -5,7 +5,9 @@
 
 
 def job_dir(settings: BaseSettings) -> Optional[str]:
-    path: str = settings["JOBDIR"]
-    if path and not Path(path).exists():
+    path: Optional[str] = settings["JOBDIR"]
+    if not path:
+        return None
+    if not Path(path).exists():
         Path(path).mkdir(parents=True)
     return path


Test patch: