Repository: schemathesis/schemathesis

Issue:
[BUG] Error with sampled_from() strategy and format()
### Checklist

- [x] I checked the [FAQ section](https://schemathesis.readthedocs.io/en/stable/faq/) of the documentation
- [x] I looked for similar issues in the [issue tracker](https://github.com/schemathesis/schemathesis/issues)
- [x] I am using the latest version of Schemathesis

### Describe the bug

My goal is to provide a strategy that sends a binary file via PUT in a pytest test using Schemathesis. To test that I tried to use the approach described in the "Quick Start: PDF File Upload" section in the documentation. Instead of passing the strategy to `schemathesis.openapi.media_type()` I pass it to `schemathesis.openapi.format()`. From what I understand from the documentation this should be valid.

When I run this, the following error is reported:

```
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\pytest\plugin.py:249: in collect
    items = [item for operation in self.schema.get_all_operations() for item in self._gen_items(operation)]
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\pytest\plugin.py:164: in _gen_items
    funcobj = create_test(
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\hypothesis\builder.py:185: in create_test
    hypothesis_test = add_coverage(
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\hypothesis\builder.py:338: in add_coverage
    for case in generate_coverage_cases(
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\hypothesis\builder.py:376: in generate_coverage_cases
    for case in _iter_coverage_cases(
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\hypothesis\builder.py:714: in _iter_coverage_cases
    elif value.generation_mode == GenerationMode.POSITIVE and not seen_positive.insert(data.kwargs):
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\coverage.py:423: in insert
    key = _to_hashable_key(value)
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\coverage.py:409: in _to_hashable_key
    serialized = _encode(value)
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\generation\coverage.py:404: in _encode
    return "".join(_iterencode(o, False))
E   TypeError: 'NoneType' object is not callable

During handling of the above exception, another exception occurred:
..\..\..\..\..\..\..\Documents\GitHub\schemathesis\src\schemathesis\pytest\plugin.py:254: in collect
    pytest.fail("Error during collection")
E   Failed: Error during collection
```

The pytest program is this:

```Python
import schemathesis
from hypothesis import strategies as st

pytest_plugins = ["schemathesis.pytest.plugin"]

# Load from file
schema = schemathesis.openapi.from_path("pdf-put-openapi.yaml")

pdf_strategy = st.sampled_from(
    [
        b"%PDF-1.4\n1 0 obj\n<<\n/Type /Catalog\n/Pages 2 0 R\n>>\nendobj\nxref\n0 3\n0000000000 65535 f \ntrailer\n<<\n/Size 3\n/Root 10 R\n>>\nstartxref\n9\n%%EOF",
        b"%PDF-1.5\n%\xe2\xe3\xcf\xd3\n1 0 obj\n<<\n/Type /Catalog\n/Pages 2 0 R\n>>\nendobj\nxref\n0 3\n0000000000 65535 f \ntrailer\n<<\n/Size 3\n/Root 1 0 R\n>>\nstartxref\n9\n%%EOF",
    ]
)

schemathesis.openapi.format("custom-pdf", pdf_strategy)

@schema.parametrize()
def test_api(case):
    response = case.call(base_url="http://192.168.1.4/api/v1", timeout=60)
    case.validate_response(response)
```

### To Reproduce

1. Save the above Python program to `test_format.py`, save the OpenAPI schema below to `pdf-put-openapi.yaml`
2. Set up a server that responds to the test requests
2. Execute `pytest .\test_format.py`

Please include a minimal API schema causing this issue:

```yaml
openapi: 3.1.0
info:
  title: Test
  version: 1.0.0

paths:
  /jobs/{jobId}}/1.unf:
    summary: Upload
    parameters:
      - name: jobId
        in: path
        required: true
        schema:
          minimum: 1
          maximum: 256
          type: integer
    put:
      operationId: putJob
      summary: Test format
      requestBody:
        required: true
        content:
          application/octet-stream:
            schema:
              type: string
              format: custom-pdf
```

### Expected behavior

The test should be executed successfully.

### Environment

```
- OS: Windows 11
- Python version: 3.12.8
- Schemathesis version: 4.4.2
- Spec version: OpenAPI 3.1.0
```

### Additional context

I noticed that if the strings are not Python `b''` binary strings but plain `""` strings the error does not occur. But in my real test I would like to use binary data that was read from a file instead of literals in the test program, with something like this:

```
def load_job_files():
    return [open(fname, "rb").read() for fname in unf_files]

unf_strategy = st.sampled_from(load_job_files())
```

The error output copied above was produced with a local clone of the Schemathesis repository. Before that I had the error also with the Schemathesis version that was installed by `pip`.

Discussion hints:

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b5a7f83ae3..aa98128b2e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@
 ### :bug: Fixed
 
 - Empty `$ref` references in schemas are now gracefully skipped during bundling.
+- Custom format strategies returning binary data. [#3289](https://github.com/schemathesis/schemathesis/issues/3289)
 
 ## [4.4.2](https://github.com/schemathesis/schemathesis/compare/v4.4.1...v4.4.2) - 2025-11-07
 
diff --git a/src/schemathesis/specs/openapi/formats.py b/src/schemathesis/specs/openapi/formats.py
index df12218197..4f0d519343 100644
--- a/src/schemathesis/specs/openapi/formats.py
+++ b/src/schemathesis/specs/openapi/formats.py
@@ -63,7 +63,13 @@ def register_string_format(name: str, strategy: st.SearchStrategy) -> None:
     if not isinstance(strategy, SearchStrategy):
         raise TypeError(f"strategy must be of type {SearchStrategy}, not {type(strategy)}")
 
-    STRING_FORMATS[name] = strategy
+    # Wrap bytes in Binary so hypothesis-jsonschema can process binary data from user-provided formats
+    def wrap_bytes(value: bytes | str | Binary) -> Binary | str:
+        if isinstance(value, bytes):
+            return Binary(value)
+        return value
+
+    STRING_FORMATS[name] = strategy.map(wrap_bytes)
 
 
 def unregister_string_format(name: str) -> None:


Test patch:
diff --git a/test/specs/openapi/test_hypothesis.py b/test/specs/openapi/test_hypothesis.py
index 895a4012bb..0f581d06bd 100644
--- a/test/specs/openapi/test_hypothesis.py
+++ b/test/specs/openapi/test_hypothesis.py
@@ -495,3 +495,50 @@ def test_unregister_string_format_valid():
 def test_unregister_string_format_invalid():
     with pytest.raises(ValueError, match="Unknown Open API format: unknown"):
         formats.unregister_string_format("unknown")
+
+
+def test_custom_format_with_bytes(testdir):
+    # See GH-3289: custom formats returning bytes should work
+    testdir.make_test(
+        """
+import schemathesis
+from hypothesis import strategies as st
+
+# Register a custom format that returns bytes
+pdf_strategy = st.sampled_from([
+    b"%PDF-1.4\\n1 0 obj\\n",
+    b"%PDF-1.5\\n%\\xe2\\xe3",
+])
+schemathesis.openapi.format("custom-pdf", pdf_strategy)
+
+schema = schemathesis.openapi.from_dict({
+    "openapi": "3.0.0",
+    "info": {"title": "Test", "version": "1.0.0"},
+    "paths": {
+        "/upload": {
+            "put": {
+                "requestBody": {
+                    "required": True,
+                    "content": {
+                        "application/octet-stream": {
+                            "schema": {
+                                "type": "string",
+                                "format": "custom-pdf"
+                            }
+                        }
+                    }
+                },
+                "responses": {"200": {"description": "OK"}}
+            }
+        }
+    }
+})
+
+@schema.parametrize()
+def test_api(case):
+    # Should not crash
+    pass
+        """,
+    )
+    result = testdir.runpytest("-v", "-s")
+    result.assert_outcomes(passed=1)