## Objective

Create a web application that scrapes and displays the top 30 stories directly from the Hacker News website (https://news.ycombinator.com/), refreshing every 3 minutes. The application should track changes in story positions, ranking them accordingly (if they moved up or down in the list since last scrape), and present a user-friendly interface with detailed information for each story. Display at most 30 stories.

## Core Requirements

1. **Web Scraping**
   - Scrape the top 30 stories from Hacker News (https://news.ycombinator.com/)
   - Use asynchronous methods for efficient scraping and updating
   - Implement rate limiting and respect robots.txt to avoid overloading the HN servers

2. **Data Management**
   - Store story details in an SQLite database
   - Fields: id, title, url, hn_url, score, poster, comments_count, time_posted, description, image_url, last_position, current_position

3. **Metadata Extraction**
   - Scrape Open Graph metadata (og:image, og:description) from story URLs
   - Implement caching to avoid repeated scraping of the same URL
   - Use a fallback system: og:image > website screenshot > placeholder image

4. **Screenshot Capture**
   - Use Selenium with Firefox (headless mode) to capture screenshots when og:image is unavailable
   - Implement a simple queue system for screenshot tasks to manage resource usage
   - Store screenshots with a naming convention that includes the story ID and timestamp

5. **Trend Tracking**
   - Compare current positions with previous positions to determine ranking changes
   - Use arrows (▲/▶/▼) to indicate position shifts next to each story's index and title

6. **User Interface**
   - Create a clean, responsive design using basic HTML and CSS
   - Display for each story:
     - Trend arrow
     - Index
     - Title (linked to original URL)
     - Resized image (og:image, screenshot, or placeholder)
     - Score
     - Comment count (linked to HN discussion)
     - Poster name
     - Time posted (in relative format, e.g., "2 hours ago")
     - URL (shortened if necessary)
     - Description (truncated if too long)

7. **Auto-Refresh**
   - Implement a server-side refresh every 3 minutes
   - Provide a visual indicator when updates are in progress
   - Allow users to manually trigger a refresh

8. **Error Handling and Logging**
   - Implement basic error handling for scraping and database operations
   - Set up simple logging to track errors and important events

## Technical Stack

- Backend: Python with Flask
- Database: SQLite with SQLAlchemy ORM
- Frontend: HTML, CSS
- Web Scraping: Beautiful Soup
- Asynchronous Tasks: Background tasks using threading or asyncio

## Project Structure

```
hn-clone/
├── app/
│   ├── __init__.py
│   ├── main.py              # Flask app initialization
│   ├── models.py            # SQLAlchemy models
│   ├── database.py          # Database connection and session management
│   ├── scraper/
│   │   ├── __init__.py
│   │   ├── hn_scraper.py    # Hacker News scraping logic
│   │   └── metadata_scraper.py  # URL metadata scraping logic
│   ├── tasks/
│   │   ├── __init__.py
│   │   ├── scrape_tasks.py  # Background tasks for scraping
│   │   └── screenshot_tasks.py  # Background tasks for taking screenshots
│   └── utils/
│       ├── __init__.py
│       └── helpers.py       # General helper functions
├── static/
│   ├── css/
│   └── images/
├── templates/
│   ├── base.html
│   └── index.html
├── .gitignore
├── README.md
└── requirements.txt
```

## Development Workflow

1. Set up the development environment with virtual environment and install dependencies
2. Implement database models
3. Develop the scraping logic
4. Create the frontend UI
5. Implement background tasks for scraping and screenshot capture
6. Integrate all components and implement the refresh functionality
7. Basic error handling and logging
8. Manual testing and refinement
9. Comment extensively

You are an expert in Python, FastAPI, and scalable API development.

Key Principles

- Write concise, technical responses with accurate Python examples.
- Use functional, declarative programming; avoid classes where possible.
- Prefer iteration and modularization over code duplication.
- Use descriptive variable names with auxiliary verbs (e.g., is_active, has_permission).
- Use lowercase with underscores for directories and files (e.g., routers/user_routes.py).
- Favor named exports for routes and utility functions.
- Use the Receive an Object, Return an Object (RORO) pattern.

Python/FastAPI

- Use def for pure functions and async def for asynchronous operations.
- Use type hints for all function signatures. Prefer Pydantic models over raw dictionaries for input validation.
- File structure: exported router, sub-routes, utilities, static content, types (models, schemas).
- Avoid unnecessary curly braces in conditional statements.
- For single-line statements in conditionals, omit curly braces.
- Use concise, one-line syntax for simple conditional statements (e.g., if condition: do_something()).

Error Handling and Validation

- Prioritize error handling and edge cases:
- Handle errors and edge cases at the beginning of functions.
- Use early returns for error conditions to avoid deeply nested if statements.
- Place the happy path last in the function for improved readability.
- Avoid unnecessary else statements; use the if-return pattern instead.
- Use guard clauses to handle preconditions and invalid states early.
- Implement proper error logging and user-friendly error messages.
- Use custom error types or error factories for consistent error handling.

Dependencies

- FastAPI
- Pydantic v2
- Async database libraries like asyncpg or aiomysql
- SQLAlchemy 2.0 (if using ORM features)

FastAPI-Specific Guidelines

- Use functional components (plain functions) and Pydantic models for input validation and response schemas.
- Use declarative route definitions with clear return type annotations.
- Use def for synchronous operations and async def for asynchronous ones.
- Minimize @app.on_event("startup") and @app.on_event("shutdown"); prefer lifespan context managers for managing startup and shutdownevents.
- Use middleware for logging, error monitoring, and performance optimization.
- Optimize for performance using async functions for I/O-bound tasks, caching strategies, and lazy loading.
- Use HTTPException for expected errors and model them as specific HTTP responses.
- Use middleware for handling unexpected errors, logging, and error monitoring.
- Use Pydantic's BaseModel for consistent input/output validation and response schemas.

Performance Optimization

- Minimize blocking I/O operations; use asynchronous operations for all database calls and external API requests.
- Implement caching for static and frequently accessed data using tools like Redis or in-memory stores.
- Optimize data serialization and deserialization with Pydantic.
- Use lazy loading techniques for large datasets and substantial API responses.

Key Conventions

1. Rely on FastAPI’s dependency injection system for managing state and shared resources.
2. Prioritize API performance metrics (response time, latency, throughput).
3. Limit blocking operations in routes:

- Favor asynchronous and non-blocking flows.
- Use dedicated async functions for database and external API operations.
- Structure routes and dependencies clearly to optimize readability and maintainability.