please analyze this spec: Okay, let's create a detailed, language-agnostic specification for the Sifreci (CCcam Line Manager) Plugin. This spec will focus on what the plugin should do, rather than how it's implemented in a specific language.

Sifreci Plugin - Language Agnostic Specification
Version: 2.0 Date: October 26, 2023

1. Overview

The Sifreci Plugin is designed to automate the discovery, testing, storage, management, and deployment of CCcam (C-lines) and potentially Newcamd (N-lines) connection parameters for use with softcam clients (e.g., OSCam, NCam) on a digital TV receiver. Its primary goal is to maintain a list of working remote servers to facilitate access to entitled services.

2. Core Modules & Responsibilities

The plugin shall consist of the following logical modules:

2.1. Scraper Module:

Responsibility: Collect potential C-lines (and N-lines) from various online sources.
Inputs: List of source configurations (URLs, parsing rules/selectors, source type).
Outputs: A list of raw, unvalidated C-line/N-line strings or structured data (server, port, user, pass, version).
Functionality:
Fetch content from HTTP/HTTPS URLs.
Support for parsing HTML, XML, JSON, and plain text content.
Utilize regular expressions and/or DOM selectors for line extraction.
Implement configurable user-agent rotation.
Implement configurable request timeouts and retries.
Support for handling basic cookies and referrers if required by sources.
Optional: Caching mechanism for fetched web content to reduce load during development or repeated runs.
Specific scrapers for different source types (e.g., forums, dedicated free-line sites, RSS feeds).
2.2. Line Tester Module:

Responsibility: Validate the connectivity and authentication of collected C-lines/N-lines.
Inputs: A single C-line or N-line (structured: server, port, user, pass, version). Configurable test timeout.
Outputs: Boolean status (success/failure), optionally additional diagnostic information (e.g., error type, latency).
Functionality:
Implement the client-side CCcam protocol handshake and authentication.
Implement the client-side Newcamd protocol handshake and authentication (Future Scope).
Handle network connection errors (timeout, host not found, connection refused).
Handle authentication failures.
Report success if authentication and basic handshake complete.
2.3. Storage Module:

Responsibility: Persistently store and manage C-lines/N-lines and associated metadata.
Interface: Provide an API for adding, retrieving, updating, and deleting line data.
Data Schema (Conceptual - for C-lines):
line_id (Primary Key, Unique Identifier)
server_hostname (String)
server_port (Integer)
username (String)
password (String)
protocol_version (String, e.g., "2.1.4")
line_protocol (Enum: CCcam, Newcamd)
source_category (String, e.g., "daily_text", "testious", "forum_X")
first_added_timestamp (Timestamp)
last_checked_timestamp (Timestamp)
last_seen_active_timestamp (Timestamp)
status (Enum: active, inactive, untested, failed_test)
is_in_use_by_softcam (Boolean, indicates if currently deployed to softcam config)
webif_confirmed_active (Boolean, indicates if softcam WebIF reported it as working)
failure_count (Integer, consecutive test failures)
priority_score (Integer, for sorting/selection)
(Composite Unique Constraint on: server_hostname, server_port, username, password, line_protocol)
Data Schema (Conceptual - for Scraper Providers/Sources):
provider_id (Primary Key)
provider_url (String, Unique)
provider_category (String, e.g., "daily", "text", "forum")
last_scraped_timestamp (Timestamp)
last_success_scrape_timestamp (Timestamp)
Data Schema (Conceptual - for Failed Line History - separate from main lines if desired for quicker checks):
server_hostname, server_port, username, password, line_protocol
last_failed_timestamp (Timestamp)
Functionality:
Efficient querying based on various criteria (status, source, server, etc.).
Prevent duplicate entries of the exact same line.
Track when lines were last tested and seen active.
2.4. Orchestration & Management Module (Manager/Cleaner):

Responsibility: Coordinate the scraping, testing, and maintenance of lines.
Functionality:
Line Collection:
Iterate through configured scraper sources.
For each newly scraped line:
Check if the exact line (or a line for the same server with same credentials) already exists in storage (active, untested, or recently failed). If so, skip or handle accordingly (e.g., update last_scraped_timestamp for provider).
Requirement 1 Adherence: If an active and webif_confirmed_active (if feature enabled) line already exists in storage for the same server hostname, add the newly scraped line to the untested pool without immediate testing.
Otherwise, pass the line to the Line Tester Module.
Store successfully tested lines as active (or active pending WebIF check). Store failed lines appropriately.
Line Maintenance (Cleaning):
Periodically re-test lines marked active or inactive from storage.
Update last_checked_timestamp.
If an active line fails, mark it inactive and update last_seen_active_timestamp. Increment failure_count.
If an inactive line passes, mark it active and update last_seen_active_timestamp. Reset failure_count.
Periodically test lines from the untested pool.
Requirement 1 Adherence: Before testing an untested line, check if an active and webif_confirmed_active (if feature enabled) line now exists for its server hostname. If so, defer testing this untested line.
Process results as with newly scraped lines.
Aging/Removal:
Remove lines that have been inactive and failed X consecutive tests or have not been seen active for Y days.
Remove lines from failed_test history after Z days to allow re-scraping/re-testing.
Remove very old untested lines.
DoS Prevention (Requirement 2):
Implement a cooldown mechanism per server hostname: after a failed test for a new line on a server, do not attempt to test another new line on the same server for a configurable period.
Implement a global rate limit between individual line tests during mass collection phases.
Softcam WebIF Integration (Requirement 3 - Part 1: Verification):
After a line passes the Line Tester, and if it's deployed to the softcam, query the softcam's Web Interface (via its API - XML/JSON).
Check if the reader corresponding to the line is "CONNECTED" or shows positive status (e.g., receiving ECMs).
Update the line's webif_confirmed_active status in storage.
2.5. Softcam Configuration Loader Module:

Responsibility: Generate and deploy configuration files for softcams.
Inputs: List of selected lines from storage. Softcam type (OSCam, NCam). Path to softcam configuration directory. Template for reader configuration.
Outputs: Updated softcam server configuration file(s).
Functionality:
Select a pool of lines from storage based on status and priority.
Requirement 3 Adherence (Part 2: Prioritization): Give highest priority to lines marked active AND webif_confirmed_active. Then active (TestCline only). Consider source_category (e.g., "daily" preferred), last_seen_active_timestamp.
Limit the total number of lines to deploy based on configuration.
Format each selected line into the appropriate reader/server syntax for the target softcam.
Write the generated configuration to the specified file(s), potentially backing up existing ones.
Update is_in_use_by_softcam status in storage for deployed lines.
Optional: Trigger a softcam restart/reload if supported and configured.
2.6. User Interface Module (Enigma2 Plugin Specific):

Responsibility: Provide user interaction within the Enigma2 environment.
Functionality:
Display plugin status and logs.
Allow manual triggering of:
Line Collection
Line Cleaning/Maintenance
Softcam Configuration Loading
Display summary statistics (e.g., total active lines, lines per server).
Configuration options (e.g., test timeouts, number of lines to load, WebIF credentials).
2.7. Scheduler/Background Runner Module:

Responsibility: Execute plugin tasks (collection, cleaning, loading) automatically on a schedule.
Functionality:
Configurable intervals for different tasks.
Ensure only one instance of major tasks runs concurrently.
3. Key Requirements Summary (Cross-Cutting)

R1 (Active Server Skip): Do not attempt to test a newly scraped/untested line if a confirmed active (and ideally webif_confirmed_active) line already exists for the same server hostname in the primary line storage. Such new lines should be added to an untested pool for later consideration.
R2 (DoS Prevention): Implement server-specific cooldowns and global rate limits when testing multiple new lines, especially for the same server, to avoid being flagged.
R3 (WebIF Verification & Prioritization):
Verification: After a line passes the internal Line Tester, attempt to verify its operational status through the softcam's Web Interface API. Store this verification status.
Prioritization: When selecting lines for softcam deployment, give higher priority to lines that are active AND webif_confirmed_active.
R4 (Regular Operations with Adherence): All regular collection and cleaning operations must respect R1, R2, and R3.
4. Configuration

The plugin shall support configuration for:

Scraper sources (URLs, types, parsing rules).
Line Tester timeouts.
DoS prevention parameters (cooldowns, rate limits).
Softcam WebIF access (IP, port, credentials, API type).
Softcam configuration paths and type.
Number of lines to deploy to softcam.
Line aging and removal thresholds.
Logging levels.
Scheduling intervals for background tasks.
5. Error Handling & Logging

Comprehensive logging for all operations, errors, and significant events.
Graceful handling of network errors, parsing errors, and unexpected API responses.
Clear error messages for the user (via UI) and logs.
6. Future Scope (Optional)

Support for Newcamd (N-lines) protocol testing and management.
More advanced analytics on line performance (e.g., ECM times, stability).
UI for browsing and manually managing stored lines.
Support for multiple softcam instances.
Automatic detection of softcam type and configuration paths.
This specification aims to be comprehensive enough to guide development regardless of the chosen programming language or specific Enigma2 environment details, while capturing the core logic and requirements you've outlined.