Repository: scrapy/scrapy
Pull request #6684 (2025-02-18 21:54:21)

curl: add support for parsing -b,--cookie
When parsing `curl` command line options, Scrapy currently supports grabbing cookie values from `-H "Cookie: ..."` parameters. However, curl also supports specifying cookies via `-b` and `--cookie`. This PR adds support for that.

I updated an existing test case to use the `-b` method to specify one of the cookies.

Motivation: Previously, the `Copy as cURL` option in Google Chrome's Developer Tools would generate a curl command with cookies specified via `-H "Cookie: ..."`. A recent update seems to have changed that feature to use `-b`.


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/6684?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
Attention: Patch coverage is `16.66667%` with `5 lines` in your changes missing coverage. Please review.
> Project coverage is 33.50%. Comparing base [(`a898331`)](https://app.codecov.io/gh/scrapy/scrapy/commit/a898331d14f889c1d4860cf1a364ba28285090a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) to head [(`a670014`)](https://app.codecov.io/gh/scrapy/scrapy/commit/a6700147b281d78053ae67918f366433628820a0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy).

| [Files with missing lines](https://app.codecov.io/gh/scrapy/scrapy/pull/6684?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Patch % | Lines |
|---|---|---|
| [scrapy/utils/curl.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6684?src=pr&el=tree&filepath=scrapy%2Futils%2Fcurl.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL2N1cmwucHk=) | 16.66% | [5 Missing :warning: ](https://app.codecov.io/gh/scrapy/scrapy/pull/6684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #6684       +/-   ##
===========================================
- Coverage   91.36%   33.50%   -57.87%     
===========================================
  Files         162      162               
  Lines       12048    12054        +6     
  Branches     1551     1554        +3     
===========================================
- Hits        11008     4039     -6969     
- Misses        735     7884     +7149     
+ Partials      305      131      -174     
```

| [Files with missing lines](https://app.codecov.io/gh/scrapy/scrapy/pull/6684?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/utils/curl.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6684?src=pr&el=tree&filepath=scrapy%2Futils%2Fcurl.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL2N1cmwucHk=) | `32.87% <16.66%> (-67.13%)` | :arrow_down: |

... and [147 files with indirect coverage changes](https://app.codecov.io/gh/scrapy/scrapy/pull/6684/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy)

</details>

Patch:
diff --git a/scrapy/utils/curl.py b/scrapy/utils/curl.py
index a563dc79a..a40ee8997 100644
--- a/scrapy/utils/curl.py
+++ b/scrapy/utils/curl.py
@@ -36,6 +36,7 @@ curl_parser = CurlParser()
 curl_parser.add_argument("url")
 curl_parser.add_argument("-H", "--header", dest="headers", action="append")
 curl_parser.add_argument("-X", "--request", dest="method")
+curl_parser.add_argument("-b", "--cookie", dest="cookies", action="append")
 curl_parser.add_argument("-d", "--data", "--data-raw", dest="data", action=DataAction)
 curl_parser.add_argument("-u", "--user", dest="auth")
 
@@ -68,6 +69,14 @@ def _parse_headers_and_cookies(
         else:
             headers.append((name, val))
 
+    for cookie_param in parsed_args.cookies or ():
+        # curl can treat this parameter as either "key=value; key2=value2" pairs, or a filename.
+        # Scrapy will only support key-value pairs.
+        if "=" not in cookie_param:
+            continue
+        for name, morsel in SimpleCookie(cookie_param).items():
+            cookies[name] = morsel.value
+
     if parsed_args.auth:
         user, password = parsed_args.auth.split(":", 1)
         headers.append(("Authorization", basic_auth_header(user, password)))


Test patch:
diff --git a/tests/test_utils_curl.py b/tests/test_utils_curl.py
index 1816db29b..5d99161bf 100644
--- a/tests/test_utils_curl.py
+++ b/tests/test_utils_curl.py
@@ -49,8 +49,8 @@ class CurlToRequestKwargsTest(unittest.TestCase):
             "ml,application/xhtml+xml,application/xml;q=0.9,image/webp,image/a"
             "png,*/*;q=0.8' -H 'Referer: http://httpbin.org/' -H 'Cookie: _gau"
             "ges_unique_year=1; _gauges_unique=1; _gauges_unique_month=1; _gau"
-            "ges_unique_hour=1; _gauges_unique_day=1' -H 'Connection: keep-ali"
-            "ve' --compressed"
+            "ges_unique_hour=1' -H 'Connection: keep-alive' --compressed -b '_"
+            "gauges_unique_day=1'"
         )
         expected_result = {
             "method": "GET",