Skill: charlie-morrison/avro-schema-manager (version 1.0.1)

--- name: cm-avro-schema-manager description: Manage Apache Avro schemas — validate structure, check forward/backward compatibility, plan schema evolution, audit namespace conventions, and generate code stubs. Works with Confluent Schema Registry, AWS Glue Schema Registry, or standalone .avsc files. Use when asked to review Avro schemas, check schema compatibility, plan schema migration, validate .avsc files, audit schema registry, or generate Avro code. Triggers on "avro", "avro schema", "schema registry", "schema compatibility", "schema evolution", ".avsc", "avsc", "confluent schema registry", "schema migration", "backward compatible", "forward compatible", "avro namespace", "serialization schema". metadata: tags: ["avro", "schema", "schema-registry", "data-engineering", "serialization", "kafka", "compatibility", "data-contracts", "streaming", "data-governance"] --- # Avro Schema Manager Manage Apache Avro schemas across your data platform. Validates schema structure, checks forward/backward compatibility for safe evolution, audits namespace conventions, reviews schema registry configuration, and generates language-specific code stubs. Acts as a senior data engineer ensuring schema quality and safe evolution. ## Usage **Basic:** `Validate the Avro schemas in /path/to/schemas/` **Focused:** `Check compatibility between schema v3 and v4` | `Review namespace conventions` | `Generate Java POJOs from this schema` | `Plan migration for adding a field` ## How It Works ### Step 1: Discover Schema Files ```bash find /path/to/project -name "*.avsc" -o -name "*.avdl" -o -name "*.avpr" grep -r "schema.registry.url" /path/to/project --include="*.properties" grep -rl "avro.schema\|AvroSchema\|fastavro" /path/to/project --include="*.py" --include="*.java" ``` Parses schema type (record, enum, fixed), namespace/name, fields with types/defaults/docs, logical types (date, timestamp, decimal, uuid), complex types (unions, arrays, maps), and cross-schema references. ### Step 2: Validate Schema Structure ``` Schema Validation: UserEvent (com.company.events.user) PASS: Valid record with namespace and documentation PASS: All fields have doc strings PASS: Logical types correct (timestamp-millis) PASS: Nullable fields use ["null", "string"] with default null WARN: event_id could use logicalType: uuid for semantic clarity WARN: Enum "EventType" is inline — extract to separate .avsc for reuse Schema Validation: OrderCreated (com.company.events.order) FAIL: Nullable field wrong union order "discount_code": ["string", "null"] FIX: ["null", "string"], default: null (null-first convention) FAIL: Missing default on nullable field "referral_source" Type ["null", "string"] but no default — old producers cause errors FIX: Add "default": null FAIL: "amount" is bytes without logical type FIX: {"type": "bytes", "logicalType": "decimal", "precision": 10, "scale": 2} FAIL: Mixed naming — "firstName" vs snake_case elsewhere FIX: "first_name" with alias: {"aliases": ["firstName"]} WARN: 4/8 fields missing documentation ``` ### Step 3: Check Schema Compatibility ``` Compatibility Analysis: UserEvent v3 -> v4 (mode: BACKWARD) 1. ADD "device_type" (string, default: "unknown") BACKWARD: YES | FORWARD: YES | FULL: YES 2. REMOVE "legacy_flag" BACKWARD: YES (consumers use default) | FORWARD: NO RISK: Consumers still reading legacy_flag get default/null ACTION: Verify no consumer depends on this field 3. MODIFY "user_id": int -> long BACKWARD: NO | FORWARD: NO | FULL: NO BREAKING CHANGE — deserialization failures guaranteed FIX: Two-phase migration: Phase 1: Add "user_id_v2" (long), keep "user_id" (int) Phase 2: Migrate all consumers to user_id_v2 Phase 3: Remove "user_id" in future version 4. ADD enum symbol "REFUND" to EventType BACKWARD: YES | FORWARD: NO RISK: Consumers without default case in switch/match will throw VERDICT: FAIL — Schema Registry will REJECT under BACKWARD mode BLOCKING: Change 3 (type widening) must be resolved first ``` ### Step 4: Audit Namespace Conventions ``` 8 namespaces found FAIL: "UserActivity" has no namespace — collision risk, default package FIX: Add namespace "com.company.events.user" FAIL: Inconsistent prefix: "events.user" vs "com.company.events.user" FIX: Standardize to com.company.<domain>.<subdomain> FAIL: "Config" uses too-broad "com.company" FIX: Use "com.company.common" or "com.company.config" FAIL: Schema "OrderItem" references "com.company.model.Product" but no .avsc file defines it — unversioned dependency FIX: Keep all schema files in repository RECOMMEND: Document namespace convention: com.company.events.* (event schemas) com.company.model.* (data model schemas) com.company.common (shared types) ``` ### Step 5: Review Schema Registry ``` Confluent Schema Registry | 34 subjects | 127 versions Global compatibility: BACKWARD WARN: "internal-logs-value" set to NONE — any change accepted RISK: Breaking changes reach prod. FIX: Set BACKWARD minimum PASS: "payment-events-value" uses BACKWARD_TRANSITIVE Checks against ALL prior versions — good for critical data FAIL: 3 subjects with non-standard naming "UserEvent" (no -value suffix), "order.created.v2" (dots + version) FIX: Standardize to TopicNameStrategy: <topic>-key, <topic>-value WARN: "order-events-value" has 8 versions — review if all consumed Soft-delete unused versions after consumer migration FAIL: No schema validation in CI/CD FIX: Add PR gate check: curl -X POST schema-registry/compatibility/subjects/<sub>/versions/latest Block merge if compatibility fails ``` ### Step 6: Plan Schema Evolution ``` Safe path for adding "payment_method" to OrderCreated: Step 1: Add as nullable with default {"name": "payment_method", "type": ["null", "string"], "default": null} Compatible in ALL modes. Old producers: consumers get null. Step 2: Register new version — compatibility check passes Step 3: Deploy producers that populate the field Step 4: (Optional future) Make required — ONLY after all producers send it DANGEROUS changes requiring special handling: Rename field: NEVER directly — use aliases instead Change type: Add new field with new type, deprecate old Remove enum symbol: NEVER — prefix with DEPRECATED_ instead Reorder fields: SAFE in Avro (name-based), may break raw byte consumers ``` ### Step 7: Code Generation Guidance ``` Java: avro-maven-plugin generates SpecificRecord classes Python: fastavro parse_schema() + writer/reader TypeScript: avsc.Type.forSchema() for encoder/decoder Go: github.com/linkedin/goavro/v2 codec from schema JSON RECOMMEND: Automate in CI/CD build step avro-tools compile schema src/main/avro target/generated Never hand-edit generated classes ``` ### Step 8: Final Report ``` # Avro Schema Management Report ## Overall Health Score: 61/100 Schema structure: 7/10 Field conventions: 5/10 Compatibility: 6/10 Namespaces: 4/10 Registry config: 6/10 Evolution practices: 5/10 CI/CD integration: 3/10 Documentation: 5/10 ## Critical Issues 1. int->long type change on user_id — BREAKING, registry rejects 2. "UserActivity" has no namespace — collision risk 3. No CI/CD compatibility validation 4. Referenced schema "Product" not in repository 5. Nullable field with wrong union order — deserialization risk ## High Priority 6. Non-standard subject naming (3 subjects) 7. "internal-logs" compatibility set to NONE 8. No code generation automation 9. Inline enums preventing cross-schema reuse 10. 30% of fields missing documentation ``` ## Output - **Per-schema validation** with field-level checks and fixes - **Compatibility matrix** for forward/backward/full between versions - **Namespace audit** for organizational consistency - **Registry review** covering config, naming, per-subject policies - **Evolution plan** with safe migration steps for requested changes - **Code generation** guidance for Java, Python, TypeScript, Go - **Health score** 0-100 with per-category breakdown ## Tips for Best Results - Point the agent at your schema directory (where .avsc files live) - Provide both old and new schema versions for compatibility checks - Specify compatibility mode needed (BACKWARD, FORWARD, FULL) - Include consumer/producer code for skew detection - Run before any schema change deployment as a gate check


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)