GitHub repository: zvdy/parsero-go
Primary language: Go
Created: 2024-10-03T16:47:16Z
Description: Parsero is a free script written in Golang which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries tell the search engines what directories or files hosted on a web server mustn't be indexed.