open-index/open-npm huggingface dataset:
npm package: @langgraph-js/crawler-mcp
Description: A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format.
Author: KonghaYao
Latest version: 1.5.3

# Crawler for LLM with MCP Support

> A universal web crawler for LLM frameworks with Model Context Protocol (MCP) integration

A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.

```json
{
    "mcpServers": {
        "langgraph-crawler": {
            "command": "npx",
            "args": ["-y", "@langgraph-js/crawler-mcp@latest"]
        }
    }
}
```

## Supported Websites

> Note: This list is continuously expanding. Feel free to contribute by adding support for more websites.

### Documentation & Development

- [x] NPM
- [x] Vitepress Base Website
- [x] Github
- [x] DEV Community (dev.to)
- [x] MDN
- [x] Medium
- [x] SearXNG Search Result
- [ ] freeCodeCamp.org
- [ ] Docker Hub
- [ ] StackOverflow
- [ ] Anthropic

### Chinese Platforms

- [x] WeChat Public Articles (微信公众号)
- [x] Juejin (稀土掘金)
- [x] The Paper (澎湃新闻)
- [x] Jiemian (界面新闻)
- [x] Huxiu (虎嗅网)
- [x] UISDC (优设网)
- [x] CNBlogs (博客园)
- [x] SSPAI (少数派)
- [x] InfoQ
- [x] CSDN
- [ ] Zhihu (知乎)

## License

Apache-2.0
npm package: @langgraph-js/open-smith
Description: Open Smith
Author: KonghaYao
Latest version: 2.4.0

# @langgraph-js/open-smith

`@langgraph-js/open-smith` is an open-source backend for `langgraph-js` that offers LangSmith-compatible tracing and observability for your AI applications. It provides a self-hostable alternative, giving developers full control over their tracing data.

## Features

-   **LangSmith Compatibility**: Designed to work seamlessly with `langgraph-js` to capture traces in a LangSmith-like format.
-   **End-to-End Tracing**: Capture and visualize the entire lifecycle of your AI runs, from user input to final output.
-   **Detailed Run Information**: Log inputs, outputs, events, errors, and metadata for each step of your AI application.
-   **Feedback Mechanism**: Collect and store feedback on individual runs to improve your models and applications.
-   **Attachment Support**: Store relevant files and data associated with specific runs.
-   **System & Thread Tracking**: Organize and filter runs by system and conversation thread for better management and analysis.
-   **Database Agnostic**: Supports both SQLite (for lightweight local development) and PostgreSQL (for production-ready deployments) through a flexible adapter pattern.

## Installation

```sh
pnpm i -D @langgraph-js/open-smith
pnpm open-smith
```

After installation, configure your LangGraph project to use this server by setting the `LANGSMITH_ENDPOINT` environment variable:

```sh
LANGSMITH_ENDPOINT="http://localhost:7765"
LANGSMITH_API_KEY="lsv2_ts" # create key in open-smith
```

GUI Admin URL:

```sh
http://localhost:7765/ui/index.html
```

### Use PG

By default, open-smith uses SQLite to store trace data. You can configure it to use PostgreSQL instead:

To use PostgreSQL, add the following to your `.env` file:

```sh
TRACE_DATABASE_URL=postgresql://postgres:postgres@localhost:5434/open_smith
MASTER_KEY=test-test
```
npm package: @langgraph-js/smith-server
Description: Open Smith Server
Author: KonghaYao
Latest version: 0.4.0

# @langgraph-js/smith-server

`@langgraph-js/smith-server` is an open-source backend for `langgraph-js` that offers LangSmith-compatible tracing and observability for your AI applications. It provides a self-hostable alternative, giving developers full control over their tracing data.

## Features

-   **LangSmith Compatibility**: Designed to work seamlessly with `langgraph-js` to capture traces in a LangSmith-like format.
-   **End-to-End Tracing**: Capture and visualize the entire lifecycle of your AI runs, from user input to final output.
-   **Detailed Run Information**: Log inputs, outputs, events, errors, and metadata for each step of your AI application.
-   **Feedback Mechanism**: Collect and store feedback on individual runs to improve your models and applications.
-   **Attachment Support**: Store relevant files and data associated with specific runs.
-   **System & Thread Tracking**: Organize and filter runs by system and conversation thread for better management and analysis.
-   **Database Agnostic**: Supports both SQLite (for lightweight local development) and PostgreSQL (for production-ready deployments) through a flexible adapter pattern.

## Installation

```sh
pnpm i -D @langgraph-js/smith-server
pnpm open-smith
```

After installation, configure your LangGraph project to use this server by setting the `LANGSMITH_ENDPOINT` environment variable:

```sh
LANGSMITH_ENDPOINT="http://localhost:7765"
```

GUI Admin URL:

```sh
http://localhost:7765/ui/index.html
```

### Use PG

By default, open-smith uses SQLite to store trace data. You can configure it to use PostgreSQL instead:

To use PostgreSQL, add the following to your `.env` file:

```sh
TRACE_DATABASE_URL=postgresql://postgres:postgres@localhost:5434/open_smith
```