Skip to content
BuilderIO
/
gpt-crawler

Type / to search

Code
Issues
28
Pull requests
10
Actions
Projects
Security
Insights
Owner avatar
gpt-crawler
Public
BuilderIO/gpt-crawler
 2 branches
 1 tag
Latest commit
@steve8708
steve8708 Merge pull request #67 from Daethyra/main
…
82b70f2
13 hours ago
Git stats
 97 commits
Files
Type
Name
Latest commit message
Commit time
.github/workflows
chore: add build step to build workflow
5 days ago
containerapp
modified config.ts to fix containerized execution
5 days ago
src
Update configSchema in src/config.ts
4 days ago
.dockerignore
modified: .dockerignore
yesterday
.gitignore
Merge pull request #58 from luissuil/main
14 hours ago
.releaserc
chore(ci): release workflow
5 days ago
Dockerfile
Continuing file reversion for sake of PR clarity
yesterday
License
Create License
16 hours ago
README.md
Add resourceExclusions to Config type
5 days ago
config.ts
cleanup
last week
package-lock.json
chore(release): 1.0.0 [skip ci]
14 hours ago
package.json
Merge pull request #67 from Daethyra/main
13 hours ago
tsconfig.json
build: use strict ts mode
5 days ago
README.md
GPT Crawler
Crawl a site to generate knowledge files to create your own custom GPT from one or multiple URLs

Gif showing the crawl run

Example
Get started
Running locally
Clone the repository
Install dependencies
Configure the crawler
Run your crawler
Alternative methods
Running in a container with Docker
Running as a CLI
Development
Upload your data to OpenAI
Create a custom GPT
Create a custom assistant
Contributing
Example
Here is a custom GPT that I quickly made to help answer questions about how to use and integrate Builder.io by simply providing the URL to the Builder docs.

This project crawled the docs and generated the file that I uploaded as the basis for the custom GPT.

Try it out yourself by asking questions about how to integrate Builder.io into a site.

Note that you may need a paid ChatGPT plan to access this feature

Get started
Running locally
Clone the repository
Be sure you have Node.js >= 16 installed.

git clone https://github.com/builderio/gpt-crawler
Install dependencies
npm i
Configure the crawler
Open config.ts and edit the url and selectors properties to match your needs.

E.g. to crawl the Builder.io docs to make our custom GPT you can use:

export const defaultConfig: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  selector: `.docs-builder-container`,
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
};
See config.ts for all available options. Here is a sample of the common configu options:

type Config = {
  /** URL to start the crawl, if sitemap is provided then it will be used instead and download all pages in the sitemap */
  url: string;
  /** Pattern to match against for links on a page to subsequently crawl */
  match: string;
  /** Selector to grab the inner text from */
  selector: string;
  /** Don't crawl more than this many pages */
  maxPagesToCrawl: number;
  /** File name for the finished data */
  outputFileName: string;
  /** Optional resources to exclude 
   * 
   * @example
   * ['png','jpg','jpeg','gif','svg','css','js','ico','woff','woff2','ttf','eot','otf','mp4','mp3','webm','ogg','wav','flac','aac','zip','tar','gz','rar','7z','exe','dmg','apk','csv','xls','xlsx','doc','docx','pdf','epub','iso','dmg','bin','ppt','pptx','odt','avi','mkv','xml','json','yml','yaml','rss','atom','swf','txt','dart','webp','bmp','tif','psd','ai','indd','eps','ps','zipx','srt','wasm','m4v','m4a','webp','weba','m4b','opus','ogv','ogm','oga','spx','ogx','flv','3gp','3g2','jxr','wdp','jng','hief','avif','apng','avifs','heif','heic','cur','ico','ani','jp2','jpm','jpx','mj2','wmv','wma','aac','tif','tiff','mpg','mpeg','mov','avi','wmv','flv','swf','mkv','m4v','m4p','m4b','m4r','m4a','mp3','wav','wma','ogg','oga','webm','3gp','3g2','flac','spx','amr','mid','midi','mka','dts','ac3','eac3','weba','m3u','m3u8','ts','wpl','pls','vob','ifo','bup','svcd','drc','dsm','dsv','dsa','dss','vivo','ivf','dvd','fli','flc','flic','flic','mng','asf','m2v','asx','ram','ra','rm','rpm','roq','smi','smil','wmf','wmz','wmd','wvx','wmx','movie','wri','ins','isp','acsm','djvu','fb2','xps','oxps','ps','eps','ai','prn','svg','dwg','dxf','ttf','fnt','fon','otf','cab']
  */
  resourceExclusions?: string[];  
};
Run your crawler
npm start
Alternative methods
Running in a container with Docker
To obtain the output.json with a containerized execution. Go into the containerapp directory. Modify the config.ts same as above, the output.jsonfile should be generated in the data folder. Note : the outputFileName property in the config.ts file in containerapp folder is configured to work with the container.

Upload your data to OpenAI
The crawl will generate a file called output.json at the root of this project. Upload that to OpenAI to create your custom assistant or custom GPT.

Create a custom GPT
Use this option for UI access to your generated knowledge that you can easily share with others

Note: you may need a paid ChatGPT plan to create and use custom GPTs right now

Go to https://chat.openai.com/
Click your name in the bottom left corner
Choose "My GPTs" in the menu
Choose "Create a GPT"
Choose "Configure"
Under "Knowledge" choose "Upload a file" and upload the file you generated
Gif of how to upload a custom GPT

Create a custom assistant
Use this option for API access to your generated knowledge that you can integrate into your product.

Go to https://platform.openai.com/assistants
Click "+ Create"
Choose "upload" and upload the file you generated
Gif of how to upload to an assistant

Contributing
Know how to make this project better? Send a PR!



Made with love by Builder.io

About
Crawl a site to generate knowledge files to create your own custom GPT from a URL

www.builder.io/blog/custom-gpt
Resources
 Readme
License
 ISC license
 Activity
Stars
 12.1k stars
Watchers
 84 watching
Forks
 849 forks
Report repository
Releases 1
v1.0.0
Latest
14 hours ago
Contributors
18
@steve8708
@guillermoscript
@marcelovicentegc
@Daethyra
@iperzic
@Umar-Azam
@justindhillon
@pipech
@nilwurtz
@highergroundstudio
@86
+ 7 contributors
Languages
TypeScript
55.9%
 
Dockerfile
22.6%
 
JavaScript
15.4%
 
Shell
6.1%
Footer
© 2023 GitHub, Inc.
Footer navigation
Terms
Privacy
Security
Status
Docs
Contact GitHub
Pricing
API
Training
Blog
About