Skill: man0l/add-directories (version 0.1.0)

--- name: add-directories description: Use when adding new AI/startup directories to directories.json from a URL, pasted text, or GitHub awesome-list. Parses, deduplicates, appends new entries, then runs classify pipeline (analyze, cleanup, browser verify). Does NOT cover submission — use submit-directories skill for that. --- # Add Directories ## Workflow 1. **Parse** the input (URL or pasted text) into a list of directories 2. **Deduplicate** against existing entries in `directories.json` 3. **Append** new entries with required fields 4. **Classify** by running the analysis and verification pipeline 5. **Discover forms** for submission targets 6. **Submit** via automation or manual browser interaction ## Step 1: Parse Input ### From URL Fetch the page and extract directory entries. Look for patterns like: - Name + URL pairs in lists, tables, or cards - Structured data (JSON-LD, markdown tables, CSV) - Repeated DOM patterns with links ### From GitHub Topics/Repos Use `gh` CLI to explore curated lists: - `gh repo clone <owner>/<repo>` to clone awesome-lists - Parse README.md for directory links (markdown link format) - Check for JSON/YAML data files with directory entries - Can also create PRs to add your product to these lists ### From Pasted Text Parse lines/rows. Common formats: - `Name - https://url.com` or `Name | https://url.com` - Markdown links: `[Name](https://url.com)` - Markdown tables with Name and URL columns - Plain URLs (one per line) — derive name from domain - CSV/TSV with headers Extract at minimum: **name** and **url** (submission or homepage). ## Step 2: Deduplicate Load `directories.json` and check each parsed entry against existing ones by: - Exact URL match (normalize: strip trailing slash, lowercase domain) - Domain match (same domain = likely duplicate) - Name match (case-insensitive) Report duplicates to the user and skip them. ## Step 3: Append New Entries For each new directory, create an entry with this structure: ```json { "categories": ["General"], "description": "", "is_active": true, "name": "Directory Name", "pricing_type": "free", "slug": "directory-name", "submission_url": "https://example.com/submit", "url": "https://example.com/submit" } ``` Field rules: - `slug`: lowercase name, spaces to hyphens, strip special chars - `submission_url` and `url`: use the submission/signup URL if available, otherwise homepage - `description`: leave empty string (will be filled later or by user) - `categories`: default `["General"]` unless context provides a category - `pricing_type`: default `"free"` unless explicitly marked paid - `is_active`: always `true` for new entries Save the updated `directories.json`. ## Step 4: Classify Run the pipeline scripts in order using the project venv at `.venv/`: ```bash # 1. HTTP-level analysis (auth, captcha, pricing signals, dead domains) .venv/bin/python analyze_directories.py # 2. Cleanup obvious failures + build browser check list .venv/bin/python cleanup_and_categorize.py # 3. Browser verification with Playwright (10 concurrent workers) .venv/bin/python browser_verify.py # 4. Deep recheck any remaining unknowns .venv/bin/python browser_verify.py --recheck-unknown ``` Each script reads/writes `directories.json`. Steps 3-4 use `browser_check_list.json` as intermediate state (generated by step 2). After completion, report the summary: how many added, and the auth/status breakdown for the new entries. ## Step 5: Discover Forms For directories that are active and have `auth_type = none` or `auth_type = email_password`: ```bash # Discover form fields on submission pages .venv/bin/python discover_forms.py ``` This visits each submission URL with Playwright, extracts form fields via DOM queries, and updates `submission_plan.json` with discovered fields and form paths. ## Step 6: Submit ### Automated Submission Configure the PRODUCT dict in `submit_directories.py` with your details (search for `YOUR_` placeholders), then: ```bash # Auto-submit to all discovered directories .venv/bin/python submit_directories.py ``` The script uses heuristic field mapping (matching field names/labels to product data) and handles file uploads for logo/screenshot. ### Manual Browser Submission (via Playwright MCP) For directories that need manual interaction (captcha, OAuth, complex forms), use the Playwright browser tools: 1. Navigate to the submission URL 2. Take a snapshot to understand the page structure 3. Fill form fields using `browser_fill_form` or `browser_type` 4. Handle OAuth flows by switching tabs when Google login popups open 5. Upload files via `browser_file_upload` 6. Click submit and verify confirmation ### GitHub PR Submissions Some directories accept submissions via GitHub PRs to awesome-lists: 1. Fork the repo: `gh repo fork <owner>/<repo>` 2. Clone and create a branch 3. Add your product entry following the repo's format 4. Push and create PR: `gh pr create` ## Notes ### Pipeline Scripts - `analyze_directories.py` uses `ThreadPoolExecutor` with plain HTTP — fast first pass - `cleanup_and_categorize.py` triages errors (dead domains, invalid URLs, Facebook groups) and builds `browser_check_list.json` - `browser_verify.py` uses async Playwright with 10 concurrent tabs; `--recheck-unknown` does a deep DOM pass on active unknowns only - `discover_forms.py` uses async Playwright with 10 concurrent tabs; extracts form field names, types, labels, and paths - `submit_directories.py` uses async Playwright with 5 concurrent tabs; heuristic field mapping with file upload support - All scripts are idempotent — safe to re-run ### Common Submission Blockers When evaluating or submitting to directories, watch for these issues: | Blocker | Frequency | How to Detect | |---|---|---| | **Paid listing required** | ~20% | Look for pricing page, Stripe/PayPal links, "$" on submit page | | **reCAPTCHA / Turnstile** | ~10% | `iframe[src*=recaptcha]` or `[data-turnstile]` elements | | **Broken captcha** | ~2% | "Invalid site key" errors, disabled submit buttons | | **Login/account required** | ~15% | Redirect to `/login` or `/register` on submit URL | | **Business email required** | ~3% | Rejects gmail/yahoo domains (e.g., SoftwareSuggest) | | **Reciprocal link required** | ~5% | Old web directories require backlink before listing | | **Newsletter-only forms** | ~10% | Page looks like submit but is actually email signup | | **Backend API broken** | ~2% | Form submits but returns GraphQL/API errors | | **Domain parked/dead** | ~8% | No content, parking page, DNS failure | | **Cloudflare blocked** | ~3% | Challenge page, 403 errors | ### Automation Tips - **Simple HTML forms** have highest auto-submit success rate - **reCAPTCHA v3** (invisible) sometimes passes; **v2** (checkbox) never does automatically - **Google Forms** are reliably automatable - **Rich text editors** (TinyMCE, Quill) need `browser_evaluate` to set content - **Cloudinary/custom upload widgets** often break automation — use manual browser - **Cross-origin OAuth popups**: Switch tabs with `browser_tabs` action to handle Google login - **Combobox/select fields**: Use `browser_click` on the dropdown, then click the option - **Multi-step forms**: Take snapshot after each step to see new fields ### Submission Plan Structure Each entry in `submission_plan.json` contains: ```json { "directory_name": "Example AI", "submission_url": "https://example.com/submit", "status": "discovered", "copy": { "title": "Product Title Variation", "description": "Product description variation for this directory." }, "discovered_fields": [...], "form_path": "form#submit-form", "credentials": { "email": "YOUR_EMAIL", "name": "YOUR_NAME", "username": "YOUR_USERNAME", "password": "YOUR_PASSWORD" } } ``` Status values: `discovered`, `submitted`, `skipped`, `skipped_paid`, `timeout`, `no_form_found`, `no_fields_matched`, `submit_timeout`, `captcha`, `cloudflare_blocked`, `domain_parked`, `skipped_login_required`, `deferred`. ### Best ROI Directory Types (for AI/SaaS products) 1. **AI tool directories** with simple forms (FutureTools, SaaSHub, AItools.inc, etc.) 2. **Startup directories** with Google Form submissions 3. **GitHub awesome-lists** accepting PRs (free, high-quality backlinks) 4. **NoCode/SaaS aggregators** (NoCodeList, NoCodeDevs) 5. **General web directories** with DA≥30 (for SEO value) ### Security Note Before pushing to GitHub, ensure all personal data is stripped: - Search for `YOUR_` placeholders in `submission_plan.json` and `submit_directories.py` - Never commit real emails, passwords, or API keys - The `.playwright-mcp/` folder may contain console logs with personal data — add to `.gitignore`


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)