Skill: panchenbo/selenium-automation-skill (version 1.0.0)

--- name: selenium-automation description: Browser automation skill using Selenium for web scraping, form filling, and UI testing. Use when Codex needs to automate browser interactions including: (1) Filling forms and submitting data, (2) Web scraping and data extraction, (3) UI testing and validation, (4) Clicking buttons and navigating pages, (5) Handling alerts and popups, (6) Taking screenshots of web pages --- # Selenium Browser Automation Skill This skill provides comprehensive browser automation capabilities using Selenium WebDriver. ## Quick Start Basic form filling example: ```python # Fill and submit a form python scripts/form_filler.py --url https://example.com/login --username testuser --password testpass ``` Web scraping example: ```python # Extract data from a webpage python scripts/web_scraper.py --url https://example.com/data --output results.csv ``` ## Installation Requirements Before using this skill, install the required dependencies: ```bash pip install selenium webdriver-manager beautifulsoup4 pandas ``` ## Supported Browsers - **Chrome**: Full support with ChromeDriver - **Firefox**: Full support with GeckoDriver - **Edge**: Full support with EdgeDriver - **Safari**: Limited support (macOS only) ## Core Scripts ### Form Filling (`scripts/form_filler.py`) Automatically fill forms and submit data: ```python python scripts/form_filler.py --url https://example.com/login --username testuser --password testpass python scripts/form_filler.py --url https://example.com/contact --name "John Doe" --email "[REDACTED_SECRET]" --message "Hello" ``` ### Web Scraper (`scripts/web_scraper.py`) Extract data from web pages: ```python python scripts/web_scraper.py --url https://example.com/products --output products.csv python scripts/web_scraper.py --url https://example.com/news --output news.json --format json ``` ### UI Tester (`scripts/ui_tester.py`) Perform UI testing and validation: ```python python scripts/ui_tester.py --url https://example.com --element "login-button" --action click python scripts/ui_tester.py --url https://example.com --element "username" --action type --text "testuser" ``` ## Usage Examples See [examples/](examples/) for comprehensive examples including: - Form filling with various input types - Web scraping with pagination - UI testing workflows - Error handling and retries - Browser configuration options ## Browser Configuration The scripts support various browser configurations: - Headless mode for background automation - Different viewport sizes - Custom user agents - Proxy support - Download directory configuration ## Error Handling Comprehensive error handling for: - Element not found - Page load timeouts - Network issues - Browser crashes - Form validation errors ## Advanced Features - **Wait strategies**: Explicit waits, implicit waits, fluent waits - **Element locators**: ID, CSS selectors, XPath, name, class - **JavaScript execution**: Run custom JavaScript in browser - **File uploads**: Handle file input fields - **Cookies management**: Get, set, and manage cookies - **Screenshots**: Capture full page or element screenshots ## Integration with Other Skills This skill can be combined with: - **browser-opener**: Open browsers programmatically - **data-processing**: Process scraped data - **file-operations**: Save and manage output files

path: scripts/form_filler.py
content: #!/usr/bin/env python3
"""
Form filling automation script using Selenium.
Supports various input types and form submission.
"""

import sys
import argparse
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.options import ArgOptions
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
from webdriver_manager.microsoft import EdgeChromiumDriverManager
from selenium.common.exceptions import (NoSuchElementException, 
                                        TimeoutException, 
                                        ElementNotInteractableException,
                                        StaleElementReferenceException)
import logging

class FormFiller:
    """Form filling automation class."""
    
    def __init__(self, browser='chrome', headless=False, timeout=30):
        self.browser = browser.lower()
        self.headless = headless
        self.timeout = timeout
        self.driver = None
        self.wait = None
        
        # Configure logging
        logging.basicConfig(level=logging.INFO)
        self.logger = logging.getLogger(__name__)
    
    def setup_driver(self):
        """Setup WebDriver based on browser type."""
        options = self._get_browser_options()
        
        if self.browser == 'chrome':
            service = Service(ChromeDriverManager().install())
            self.driver = webdriver.Chrome(service=service, options=options)
        elif self.browser == 'firefox':
            service = Service(GeckoDriverManager().install())
            self.driver = webdriver.Firefox(service=service, options=options)
        elif self.browser == 'edge':
            service = Service(EdgeChromiumDriverManager().install())
            self.driver = webdriver.Edge(service=service, options=options)
        else:
            raise ValueError(f"Unsupported browser: {self.browser}")
        
        self.wait = WebDriverWait(self.driver, self.timeout)
        self.logger.info(f"Browser {self.browser} initialized successfully")
    
    def _get_browser_options(self):
        """Get browser-specific options."""
        if self.browser == 'chrome':
            options = Options()
            if self.headless:
                options.add_argument('--headless')
            options.add_argument('--disable-gpu')
            options.add_argument('--no-sandbox')
            options.add_argument('--disable-dev-shm-usage')
            options.add_argument('--window-size=1920,1080')
        elif self.browser == 'firefox':
            options = ArgOptions()
            if self.headless:
                options.add_argument('--headless')
            options.add_argument('--width=1920')
            options.add_argument('--height=1080')
        elif self.browser == 'edge':
            options = ArgOptions()
            if self.headless:
                options.add_argument('--headless')
            options.add_argument('--disable-gpu')
            options.add_argument('--no-sandbox')
            options.add_argument('--disable-dev-shm-usage')
            options.add_argument('--window-size=1920,1080')
        else:
            raise ValueError(f"Unsupported browser: {self.browser}")
        
        return options
    
    def fill_form(self, url, form_data, submit=True, submit_button=None):
        """
        Fill a form with provided data.
        
        Args:
            url (str): URL of the form page
            form_data (dict): Dictionary of field_name -> value pairs
            submit (bool): Whether to submit the form after filling
            submit_button (str): CSS selector or ID of submit button
        """
        try:
            self.driver.get(url)
            self.logger.info(f"Loaded page: {url}")
            
            # Wait for page to load
            self.wait.until(EC.presence_of_element_located((By.TAG_NAME, 'body')))
            
            # Fill each field
            for field_name, value in form_data.items():
                self._fill_field(field_name, value)
            
            self.logger.info("Form filled successfully")
            
            # Submit form if requested
            if submit:
                self._submit_form(submit_button)
            
            return True
            
        except Exception as e:
            self.logger.error(f"Error filling form: {str(e)}")
            return False
    
    def _fill_field(self, field_name, value):
        """Fill a single field."""
        attempts = 0
        max_attempts = 3
        
        while attempts < max_attempts:
            try:
                # Try multiple locator strategies
                element = self._find_element(field_name)
                
                if element:
                    # Clear existing value
                    element.clear()
                    
                    # Type new value
                    element.send_keys(value)
                    self.logger.info(f"Filled field '{field_name}' with '{value}'")
                    return
                
            except (NoSuchElementException, ElementNotInteractableException) as e:
                attempts += 1
                self.logger.warning(f"Attempt {attempts} failed for field '{field_name}': {str(e)}")
                if attempts < max_attempts:
                    time.sleep(1)  # Wait before retry
            
            except StaleElementReferenceException:
                attempts += 1
                self.logger.warning(f"Stale element reference for field '{field_name}', retrying...")
                if attempts < max_attempts:
                    time.sleep(1)
        
        raise NoSuchElementException(f"Could not find or interact with field: {field_name}")
    
    def _find_element(self, field_name):
        """Find an element using multiple strategies."""
        strategies = [
            (By.ID, field_name),
            (By.NAME, field_name),
            (By.CSS_SELECTOR, f"[name='{field_name}']"),
            (By.CSS_SELECTOR, f"#{field_name}"),
            (By.XPATH, f"//*[@name='{field_name}']"),
            (By.XPATH, f"//*[@id='{field_name}']"),
            (By.CSS_SELECTOR, f"[placeholder*='{field_name}']"),
            (By.CSS_SELECTOR, f"[aria-label*='{field_name}']"),
            (By.CSS_SELECTOR, f"[title*='{field_name}']"),
        ]
        
        for by, value in strategies:
            try:
                element = self.wait.until(EC.presence_of_element_located((by, value)))
                if element.is_displayed() and element.is_enabled():
                    return element
            except:
                continue
        
        return None
    
    def _submit_form(self, submit_button=None):
        """Submit the form."""
        try:
            if submit_button:
                # Use specific submit button
                button = self.wait.until(EC.element_to_be_clickable(
                    (By.CSS_SELECTOR, submit_button) if submit_button.startswith('.') else (By.ID, submit_button)
                ))
            else:
                # Try common submit button selectors
                button_selectors = [
                    'input[type="submit"]',
                    'button[type="submit"]',
                    'input[value*="submit"]',
                    'input[value*="Submit"]',
                    'button:contains("Submit")',
                    '.submit',
                    '#submit'
                ]
                
                button = None
                for selector in button_selectors:
                    try:
                        button = self.wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, selector)))
                        if button:
                            break
                    except:
                        continue
            
            if button:
                button.click()
                self.logger.info("Form submitted successfully")
                
                # Wait for page to load after submission
                time.sleep(2)
                return True
            else:
                self.logger.warning("No submit button found")
                return False
                
        except Exception as e:
            self.logger.error(f"Error submitting form: {str(e)}")
            return False
    
    def take_screenshot(self, filename='screenshot.png'):
        """Take a screenshot of the current page."""
        try:
            self.driver.save_screenshot(filename)
            self.logger.info(f"Screenshot saved as {filename}")
            return True
        except Exception as e:
            self.logger.error(f"Error taking screenshot: {str(e)}")
            return False
    
    def close(self):
        """Close the browser."""
        if self.driver:
            self.driver.quit()
            self.logger.info("Browser closed")

def main():
    """Main function to handle command line arguments."""
    parser = argparse.ArgumentParser(description='Automatically fill web forms')
    parser.add_argument('--url', required=True, help='URL of the form page')
    parser.add_argument('--browser', default='chrome', choices=['chrome', 'firefox', 'edge'],
                       help='Browser to use (default: chrome)')
    parser.add_argument('--headless', action='store_true', 
                       help='Run browser in headless mode')
    parser.add_argument('--timeout', type=int, default=30,
                       help='Timeout in seconds (default: 30)')
    parser.add_argument('--screenshot', action='store_true',
                       help='Take screenshot after filling form')
    parser.add_argument('--no-submit', action='store_true',
                       help='Do not submit the form after filling')
    parser.add_argument('--submit-button', help='CSS selector or ID of submit button')
    
    # Form data arguments
    parser.add_argument('--username', help='Username field value')
    parser.add_argument('--password', help='Password field value')
    parser.add_argument('--email', help='Email field value')
    parser.add_argument('--name', help='Name field value')
    parser.add_argument('--message', help='Message field value')
    parser.add_argument('--phone', help='Phone field value')
    parser.add_argument('--company', help='Company field value')
    
    args = parser.parse_args()
    
    # Build form data dictionary
    form_data = {}
    for field in ['username', 'password', 'email', 'name', 'message', 'phone', 'company']:
        value = getattr(args, field)
        if value:
            form_data[field] = value
    
    if not form_data:
        print("Error: No form data provided. Use --username, --password, etc.")
        sys.exit(1)
    
    # Initialize form filler
    filler = FormFiller(browser=args.browser, headless=args.headless, timeout=args.timeout)
    
    try:
        # Setup driver
        filler.setup_driver()
        
        # Fill form
        success = filler.fill_form(
            url=args.url,
            form_data=form_data,
            submit=not args.no_submit,
            submit_button=args.submit_button
        )
        
        if success:
            print("Form filled successfully!")
            
            if args.screenshot:
                filler.take_screenshot()
                print("Screenshot saved!")
        else:
            print("Failed to fill form")
            sys.exit(1)
            
    except Exception as e:
        print(f"Error: {str(e)}")
        sys.exit(1)
    finally:
        filler.close()

if __name__ == '__main__':
    main()

path: scripts/time_logger.py
content: #!/usr/bin/env python3
"""
Time logging automation script for task time registration.
"""

import sys
import argparse
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from selenium.common.exceptions import (NoSuchElementException, 
                                        TimeoutException, 
                                        ElementNotInteractableException,
                                        StaleElementReferenceException)
import logging

class TimeLogger:
    """Time logging automation class."""
    
    def __init__(self, headless=False, timeout=30):
        self.headless = headless
        self.timeout = timeout
        self.driver = None
        self.wait = None
        
        # Configure logging
        logging.basicConfig(level=logging.INFO)
        self.logger = logging.getLogger(__name__)
    
    def setup_driver(self):
        """Setup Chrome WebDriver."""
        options = Options()
        if self.headless:
            options.add_argument('--headless')
        options.add_argument('--disable-gpu')
        options.add_argument('--no-sandbox')
        options.add_argument('--disable-dev-shm-usage')
        options.add_argument('--window-size=1920,1080')
        
        service = Service(ChromeDriverManager().install())
        self.driver = webdriver.Chrome(service=service, options=options)
        self.wait = WebDriverWait(self.driver, self.timeout)
        self.logger.info("Chrome browser initialized successfully")
    
    def log_time(self, task_url, hours, description, date=None):
        """
        Log time for a task.
        
        Args:
            task_url (str): URL of the task page
            hours (str): Hours to log (e.g., "2")
            description (str): Description of the work
            date (str): Date in YYYY-MM-DD format (optional)
        """
        try:
            self.driver.get(task_url)
            self.logger.info(f"Loaded task page: {task_url}")
            
            # Wait for page to load
            self.wait.until(EC.presence_of_element_located((By.TAG_NAME, 'body')))
            
            # Take initial screenshot
            self.take_screenshot('initial_page.png')
            
            # Try to find and click time logging button
            time_logged = self._find_and_click_time_button()
            
            if time_logged:
                # Fill time logging form
                self._fill_time_form(hours, description, date)
                
                # Submit the form
                self._submit_time_form()
                
                return True
            else:
                self.logger.error("Could not find time logging button")
                return False
                
        except Exception as e:
            self.logger.error(f"Error logging time: {str(e)}")
            return False
    
    def _find_and_click_time_button(self):
        """Find and click the time logging button."""
        # Try different selectors for time logging buttons
        button_selectors = [
            'button:contains("Log Time")',
            'button:contains("Add Time")',
            'button:contains("记录时间")',
            'button:contains("登记工时")',
            'button:contains("工时登记")',
            '.log-time',
            '.add-time',
            '.time-log',
            '[title*="time"]',
            '[title*="Time"]',
            '[title*="工时"]',
            'a:contains("Time")',
            'a:contains("时间")',
            'a:contains("工时")'
        ]
        
        for selector in button_selectors:
            try:
                # Wait for button to be clickable
                button = self.wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, selector)))
                button.click()
                self.logger.info(f"Clicked time logging button: {selector}")
                
                # Wait for form to appear
                time.sleep(2)
                return True
                
            except:
                continue
        
        # Try to find by text content using XPath
        xpath_selectors = [
            '//button[contains(text(), "Time")]',
            '//button[contains(text(), "时间")]',
            '//button[contains(text(), "工时")]',
            '//a[contains(text(), "Time")]',
            '//a[contains(text(), "时间")]',
            '//a[contains(text(), "工时")]'
        ]
        
        for xpath in xpath_selectors:
            try:
                button = self.wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
                button.click()
                self.logger.info(f"Clicked time logging button with XPath: {xpath}")
                
                # Wait for form to appear
                time.sleep(2)
                return True
                
            except:
                continue
        
        return False
    
    def _fill_time_form(self, hours, description, date=None):
        """Fill the time logging form."""
        try:
            # Try to find hours input field
            hours_selectors = [
                'input[name*="hour"]',
                'input[name*="time"]',
                'input[name*="duration"]',
                'input[name*="工时"]',
                'input[name*="小时"]',
                'input[placeholder*="hour"]',
                'input[placeholder*="time"]',
                'input[placeholder*="duration"]',
                'input[placeholder*="工时"]',
                'input[placeholder*="小时"]',
                '#hours',
                '#time',
                '#duration',
                '.hours',
                '.time',
                '.duration'
            ]
            
            hours_filled = False
            for selector in hours_selectors:
                try:
                    hours_input = self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, selector)))
                    hours_input.clear()
                    hours_input.send_keys(hours)
                    self.logger.info(f"Filled hours: {hours}")
                    hours_filled = True
                    break
                except:
                    continue
            
            if not hours_filled:
                self.logger.warning("Could not find hours input field")
            
            # Try to find description textarea
            desc_selectors = [
                'textarea[name*="description"]',
                'textarea[name*="comment"]',
                'textarea[name*="note"]',
                'textarea[name*="描述"]',
                'textarea[name*="备注"]',
                'textarea[placeholder*="description"]',
                'textarea[placeholder*="comment"]',
                'textarea[placeholder*="note"]',
                'textarea[placeholder*="描述"]',
                'textarea[placeholder*="备注"]',
                '#description',
                '#comment',
                '#note',
                '.description',
                '.comment',
                '.note'
            ]
            
            desc_filled = False
            for selector in desc_selectors:
                try:
                    desc_input = self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, selector)))
                    desc_input.clear()
                    desc_input.send_keys(description)
                    self.logger.info(f"Filled description: {description}")
                    desc_filled = True
                    break
                except:
                    continue
            
            if not desc_filled:
                self.logger.warning("Could not find description textarea")
            
            # Try to find date input if provided
            if date:
                date_selectors = [
                    'input[name*="date"]',
                    'input[name*="Date"]',
                    'input[name*="日期"]',
                    'input[placeholder*="date"]',
                    'input[placeholder*="Date"]',
                    'input[placeholder*="日期"]',
                    '#date',
                    '#Date',
                    '.date'
                ]
                
                date_filled = False
                for selector in date_selectors:
                    try:
                        date_input = self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, selector)))
                        date_input.clear()
                        date_input.send_keys(date)
                        self.logger.info(f"Filled date: {date}")
                        date_filled = True
                        break
                    except:
                        continue
                
                if not date_filled:
                    self.logger.warning("Could not find date input field")
            
            # Take screenshot after filling form
            self.take_screenshot('filled_form.png')
            
        except Exception as e:
            self.logger.error(f"Error filling time form: {str(e)}")
    
    def _submit_time_form(self):
        """Submit the time logging form."""
        try:
            # Try to find submit button
            submit_selectors = [
                'input[type="submit"]',
                'button[type="submit"]',
                'input[value*="submit"]',
                'input[value*="Submit"]',
                'input[value*="保存"]',
                'input[value*="提交"]',
                'button:contains("Submit")',
                'button:contains("保存")',
                'button:contains("提交")',
                '.submit',
                '#submit',
                '.save',
                '.save-button'
            ]
            
            for selector in submit_selectors:
                try:
                    submit_button = self.wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, selector)))
                    submit_button.click()
                    self.logger.info("Clicked submit button")
                    
                    # Wait for submission to complete
                    time.sleep(3)
                    return True
                    
                except:
                    continue
            
            self.logger.warning("Could not find submit button")
            return False
            
        except Exception as e:
            self.logger.error(f"Error submitting time form: {str(e)}")
            return False
    
    def take_screenshot(self, filename='screenshot.png'):
        """Take a screenshot of the current page."""
        try:
            self.driver.save_screenshot(filename)
            self.logger.info(f"Screenshot saved as {filename}")
            return True
        except Exception as e:
            self.logger.error(f"Error taking screenshot: {str(e)}")
            return False
    
    def close(self):
        """Close the browser."""
        if self.driver:
            self.driver.quit()
            self.logger.info("Browser closed")

def main():
    """Main function to handle command line arguments."""
    parser = argparse.ArgumentParser(description='Automatically log time for tasks')
    parser.add_argument('--url', required=True, help='URL of the task page')
    parser.add_argument('--hours', required=True, help='Hours to log (e.g., "2")')
    parser.add_argument('--description', required=True, help='Description of the work')
    parser.add_argument('--date', help='Date in YYYY-MM-DD format (optional)')
    parser.add_argument('--headless', action='store_true', 
                       help='Run browser in headless mode')
    parser.add_argument('--timeout', type=int, default=30,
                       help='Timeout in seconds (default: 30)')
    parser.add_argument('--screenshot', action='store_true',
                       help='Take screenshots during the process')
    
    args = parser.parse_args()
    
    # Initialize time logger
    logger = TimeLogger(headless=args.headless, timeout=args.timeout)
    
    try:
        # Setup driver
        logger.setup_driver()
        
        # Log time
        success = logger.log_time(
            task_url=args.url,
            hours=args.hours,
            description=args.description,
            date=args.date
        )
        
        if success:
            print("Time logged successfully!")
            if args.screenshot:
                logger.take_screenshot('final_page.png')
                print("Final screenshot saved!")
        else:
            print("Failed to log time")
            sys.exit(1)
            
    except Exception as e:
        print(f"Error: {str(e)}")
        sys.exit(1)
    finally:
        logger.close()

if __name__ == '__main__':
    main()

path: scripts/web_scraper.py
content: #!/usr/bin/env python3
"""
Web scraping automation script using Selenium and BeautifulSoup.
"""

import sys
import argparse
import time
import json
import csv
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.firefox.options import FirefoxOptions
from selenium.webdriver.edge.options import EdgeOptions
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.firefox import GeckoDriverManager
from webdriver_manager.microsoft import EdgeChromiumDriverManager
from bs4 import BeautifulSoup
from selenium.common.exceptions import (NoSuchElementException, 
                                        TimeoutException,
                                        WebDriverException)
import logging

class WebScraper:
    """Web scraping automation class."""
    
    def __init__(self, browser='chrome', headless=True, timeout=30):
        self.browser = browser.lower()
        self.headless = headless
        self.timeout = timeout
        self.driver = None
        self.wait = None
        self.soup = None
        
        # Configure logging
        logging.basicConfig(level=logging.INFO)
        self.logger = logging.getLogger(__name__)
    
    def setup_driver(self):
        """Setup WebDriver based on browser type."""
        options = self._get_browser_options()
        
        if self.browser == 'chrome':
            service = Service(ChromeDriverManager().install())
            self.driver = webdriver.Chrome(service=service, options=options)
        elif self.browser == 'firefox':
            service = Service(GeckoDriverManager().install())
            self.driver = webdriver.Firefox(service=service, options=options)
        elif self.browser == 'edge':
            service = Service(EdgeChromiumDriverManager().install())
            self.driver = webdriver.Edge(service=service, options=options)
        else:
            raise ValueError(f"Unsupported browser: {self.browser}")
        
        self.wait = WebDriverWait(self.driver, self.timeout)
        self.logger.info(f"Browser {self.browser} initialized successfully")
    
    def _get_browser_options(self):
        """Get browser-specific options."""
        if self.browser == 'chrome':
            options = Options()
            if self.headless:
                options.add_argument('--headless')
            options.add_argument('--disable-gpu')
            options.add_argument('--no-sandbox')
            options.add_argument('--disable-dev-shm-usage')
            options.add_argument('--window-size=1920,1080')
        elif self.browser == 'firefox':
            options = FirefoxOptions()
            if self.headless:
                options.add_argument('--headless')
            options.add_argument('--width=1920')
            options.add_argument('--height=1080')
        elif self.browser == 'edge':
            options = EdgeOptions()
            if self.headless:
                options.add_argument('--headless')
            options.add_argument('--disable-gpu')
            options.add_argument('--no-sandbox')
            options.add_argument('--disable-dev-shm-usage')
            options.add_argument('--window-size=1920,1080')
        else:
            raise ValueError(f"Unsupported browser: {self.browser}")
        
        return options
    
    def scrape_page(self, url, selectors=None, wait_for=None):
        """
        Scrape data from a webpage.
        
        Args:
            url (str): URL to scrape
            selectors (dict): Dictionary of data_name -> CSS selector pairs
            wait_for (str): CSS selector to wait for before scraping
        """
        try:
            self.driver.get(url)
            self.logger.info(f"Loaded page: {url}")
            
            # Wait for page to load
            self.wait.until(EC.presence_of_element_located((By.TAG_NAME, 'body')))
            
            # Wait for specific element if provided
            if wait_for:
                self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, wait_for)))
            
            # Get page source and parse with BeautifulSoup
            self.soup = BeautifulSoup(self.driver.page_source, 'html.parser')
            
            # Scrape data using selectors
            if selectors:
                data = self._scrape_with_selectors(selectors)
            else:
                data = self._scrape_all_text()
            
            return data
            
        except Exception as e:
            self.logger.error(f"Error scraping page: {str(e)}")
            return None
    
    def _scrape_with_selectors(self, selectors):
        """Scrape data using CSS selectors."""
        data = {}
        
        for data_name, selector in selectors.items():
            try:
                elements = self.soup.select(selector)
                
                if elements:
                    if len(elements) == 1:
                        data[data_name] = self._extract_element_text(elements[0])
                    else:
                        data[data_name] = [self._extract_element_text(el) for el in elements]
                else:
                    data[data_name] = None
                    
            except Exception as e:
                self.logger.warning(f"Error scraping {data_name}: {str(e)}")
                data[data_name] = None
        
        return data
    
    def _extract_element_text(self, element):
        """Extract text from an element."""
        text = element.get_text(strip=True)
        
        # Extract specific attributes if needed
        if element.name == 'a':
            href = element.get('href')
            if href:
                text = f"{text} ({href})"
        
        if element.name in ['img', 'source']:
            src = element.get('src') or element.get('data-src')
            if src:
                text = f"[Image: {src}]"
        
        return text
    
    def _scrape_all_text(self):
        """Scrape all text content from the page."""
        text_content = self.soup.get_text(strip=True)
        return {'all_text': text_content}
    
    def scrape_table(self, table_selector, output_file=None, format='csv'):
        """
        Scrape data from an HTML table.
        
        Args:
            table_selector (str): CSS selector for the table
            output_file (str): Output file path
            format (str): Output format ('csv' or 'json')
        """
        try:
            table = self.soup.select_one(table_selector)
            if not table:
                raise NoSuchElementException(f"Table not found with selector: {table_selector}")
            
            # Extract table headers
            headers = []
            header_row = table.select_one('thead tr') or table.select_one('tr')
            if header_row:
                headers = [th.get_text(strip=True) for th in header_row.select('th, td')]
            
            # Extract table rows
            rows = []
            for row in table.select('tr'):
                if row == header_row:
                    continue  # Skip header row
                
                cells = [td.get_text(strip=True) for td in row.select('td, th')]
                if cells:  # Only add non-empty rows
                    rows.append(cells)
            
            # Create DataFrame
            df = pd.DataFrame(rows, columns=headers if headers else None)
            
            # Save to file if requested
            if output_file:
                if format == 'csv':
                    df.to_csv(output_file, index=False)
                elif format == 'json':
                    df.to_json(output_file, orient='records', indent=2)
                
                self.logger.info(f"Table data saved to {output_file}")
            
            return df
            
        except Exception as e:
            self.logger.error(f"Error scraping table: {str(e)}")
            return None
    
    def scrape_links(self, link_selector=None, output_file=None):
        """
        Scrape links from the page.
        
        Args:
            link_selector (str): CSS selector for links (default: all links)
            output_file (str): Output file path
        """
        try:
            if link_selector:
                links = self.soup.select(link_selector)
            else:
                links = self.soup.select('a[href]')
            
            link_data = []
            for link in links:
                href = link.get('href')
                text = link.get_text(strip=True)
                
                if href and text:
                    link_data.append({
                        'text': text,
                        'url': href,
                        'full_url': self._make_absolute_url(href)
                    })
            
            # Save to file if requested
            if output_file:
                with open(output_file, 'w', encoding='utf-8') as f:
                    json.dump(link_data, f, indent=2, ensure_ascii=False)
                self.logger.info(f"Links saved to {output_file}")
            
            return link_data
            
        except Exception as e:
            self.logger.error(f"Error scraping links: {str(e)}")
            return None
    
    def _make_absolute_url(self, url):
        """Convert relative URL to absolute URL."""
        if url.startswith(('http://', 'https://')):
            return url
        else:
            # This is a simplified version - in production, you'd use urllib.parse
            return url
    
    def take_screenshot(self, filename='screenshot.png'):
        """Take a screenshot of the current page."""
        try:
            self.driver.save_screenshot(filename)
            self.logger.info(f"Screenshot saved as {filename}")
            return True
        except Exception as e:
            self.logger.error(f"Error taking screenshot: {str(e)}")
            return False
    
    def close(self):
        """Close the browser."""
        if self.driver:
            self.driver.quit()
            self.logger.info("Browser closed")

def main():
    """Main function to handle command line arguments."""
    parser = argparse.ArgumentParser(description='Automatically scrape web pages')
    parser.add_argument('--url', required=True, help='URL to scrape')
    parser.add_argument('--browser', default='chrome', choices=['chrome', 'firefox', 'edge'],
                       help='Browser to use (default: chrome)')
    parser.add_argument('--headless', action='store_true', default=True,
                       help='Run browser in headless mode')
    parser.add_argument('--timeout', type=int, default=30,
                       help='Timeout in seconds (default: 30)')
    parser.add_argument('--output', help='Output file path')
    parser.add_argument('--format', choices=['csv', 'json', 'txt'], default='csv',
                       help='Output format (default: csv)')
    parser.add_argument('--screenshot', action='store_true',
                       help='Take screenshot of the page')
    parser.add_argument('--wait-for', help='CSS selector to wait for before scraping')
    
    # Selectors for specific data extraction
    parser.add_argument('--title-selector', default='title',
                       help='CSS selector for page title')
    parser.add_argument('--content-selector', 
                       help='CSS selector for main content')
    parser.add_argument('--table-selector', 
                       help='CSS selector for table to scrape')
    parser.add_argument('--link-selector', 
                       help='CSS selector for links to scrape')
    
    args = parser.parse_args()
    
    # Initialize web scraper
    scraper = WebScraper(browser=args.browser, headless=args.headless, timeout=args.timeout)
    
    try:
        # Setup driver
        scraper.setup_driver()
        
        # Define selectors
        selectors = {}
        if args.title_selector:
            selectors['title'] = args.title_selector
        if args.content_selector:
            selectors['content'] = args.content_selector
        
        # Scrape page
        data = scraper.scrape_page(
            url=args.url,
            selectors=selectors,
            wait_for=args.wait_for
        )
        
        if data:
            print("Scraping completed successfully!")
            
            # Process table if specified
            if args.table_selector:
                table_data = scraper.scrape_table(
                    table_selector=args.table_selector,
                    output_file=args.output,
                    format=args.format
                )
                if table_data is not None:
                    print(f"Table scraped with {len(table_data)} rows")
            
            # Process links if specified
            elif args.link_selector:
                link_data = scraper.scrape_links(
                    link_selector=args.link_selector,
                    output_file=args.output
                )
                if link_data:
                    print(f"Found {len(link_data)} links")
            
            # Process general data
            elif data:
                if args.output:
                    if args.format == 'json':
                        with open(args.output, 'w', encoding='utf-8') as f:
                            json.dump(data, f, indent=2, ensure_ascii=False)
                    elif args.format == 'csv':
                        df = pd.DataFrame([data])
                        df.to_csv(args.output, index=False)
                    elif args.format == 'txt':
                        with open(args.output, 'w', encoding='utf-8') as f:
                            for key, value in data.items():
                                f.write(f"{key}: {value}\n")
                    
                    print(f"Data saved to {args.output}")
                else:
                    print("Scraped data:")
                    for key, value in data.items():
                        print(f"{key}: {value}")
            
            # Take screenshot if requested
            if args.screenshot:
                scraper.take_screenshot()
                print("Screenshot saved!")
        else:
            print("Failed to scrape page")
            sys.exit(1)
            
    except Exception as e:
        print(f"Error: {str(e)}")
        sys.exit(1)
    finally:
        scraper.close()

if __name__ == '__main__':
    main()

Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)