Package: recap-core
What is Recap?Recap reads and writes schemas from web services, databases, and schema registries in a standard format.⭐️If you like this project, please give it a star! It helps the project get more visibility.Table of ContentsWhat is Recap?Supported FormatsInstallUsageCLIGatewayRegistryAPIDockerSchemaDocumentationSupported FormatsFormatReadWriteAvro✅✅Protobuf✅✅JSON Schema✅✅Snowflake✅PostgreSQL✅MySQL✅BigQuery✅Confluent Schema Registry✅Hive Metastore✅InstallInstall Recap and all of its optional dependencies:pipinstall'recap-core[all]'You can also select specific dependencies:pipinstall'recap-core[avro,kafka]'Seepyproject.tomlfor a list of optional dependencies.UsageCLIRecap comes with a command line interface that can list and read schemas from external systems.List the children of a URL:recaplspostgresql://user:pass@host:port/testdb["pg_toast","pg_catalog","public","information_schema"]Keep drilling down:recaplspostgresql://user:pass@host:port/testdb/public["test_types"]Read the schema for thetest_typestable as a Recap struct:recapschemapostgresql://user:pass@host:port/testdb/public/test_types{"type":"struct","fields":[{"type":"int64","name":"test_bigint","optional":true}]}GatewayRecap comes with a stateless HTTP/JSON gateway that can list and read schemas from data catalogs and databases.Start the server athttp://localhost:8000:recapserveList the schemas in a PostgreSQL database:curlhttp://localhost:8000/gateway/ls/postgresql://user:pass@host:port/testdb["pg_toast","pg_catalog","public","information_schema"]And read a schema:curlhttp://localhost:8000/gateway/schema/postgresql://user:pass@host:port/testdb/public/test_types{"type":"struct","fields":[{"type":"int64","name":"test_bigint","optional":true}]}The gateway fetches schemas from external systems in realtime and returns them as Recap schemas.An OpenAPI schema is available athttp://localhost:8000/docs.RegistryYou can store schemas in Recap's schema registry.Start the server athttp://localhost:8000:recapservePut a schema in the registry:curl-XPOST\-H"Content-Type: application/x-recap+json"\-d'{"type":"struct","fields":[{"type":"int64","name":"test_bigint","optional":true}]}'\http://localhost:8000/registry/some_schemaGet the schema (and version) from the registry:curlhttp://localhost:8000/registry/some_schema[{"type":"struct","fields":[{"type":"int64","name":"test_bigint","optional":true}]},1]Put a new version of the schema in the registry:curl-XPOST\-H"Content-Type: application/x-recap+json"\-d'{"type":"struct","fields":[{"type":"int32","name":"test_int","optional":true}]}'\http://localhost:8000/registry/some_schemaList schema versions:curlhttp://localhost:8000/registry/some_schema/versions[1,2]Get a specific version of the schema:curlhttp://localhost:8000/registry/some_schema/versions/1[{"type":"struct","fields":[{"type":"int64","name":"test_bigint","optional":true}]},1]The registry usesfsspecto store schemas in a variety of filesystems like S3, GCS, ABS, and the local filesystem. See theregistrydocs for more details.An OpenAPI schema is available athttp://localhost:8000/docs.APIRecap hasrecap.convertersandrecap.clientspackages.Converters convert schemas to and from Recap schemas.Clients read schemas from external systems (databases, schema registries, and so on) and use converters to return Recap schemas.Read a schema from PostgreSQL:fromrecap.clientsimportcreate_clientwithcreate_client("postgresql://user:pass@host:port/testdb")asc:c.schema("testdb","public","test_types")Convert the schema to Avro, Protobuf, and JSON schemas:fromrecap.converters.avroimportAvroConverterfromrecap.converters.protobufimportProtobufConverterfromrecap.converters.json_schemaimportJSONSchemaConverteravro_schema=AvroConverter().from_recap(struct)protobuf_schema=ProtobufConverter().from_recap(struct)json_schema=JSONSchemaConverter().from_recap(struct)Transpile schemas from one format to another:fromrecap.converters.json_schemaimportJSONSchemaConverterfromrecap.converters.avroimportAvroConverterjson_schema="""{"type": "object","$id": "https://recap.build/person.schema.json","properties": {"name": {"type": "string"}}}"""# Use Recap as an intermediate format to convert JSON schema to Avrostruct=JSONSchemaConverter().to_recap(json_schema)avro_schema=AvroConverter().from_recap(struct)Store schemas in Recap's schema registry:fromrecap.storage.registryimportRegistryStoragefromrecap.typesimportStructType,IntTypestorage=RegistryStorage("file:///tmp/recap-registry-storage")version=storage.put("postgresql://localhost:5432/testdb/public/test_table",StructType(fields=[IntType(32)]))storage.get("postgresql://localhost:5432/testdb/public/test_table")# Get all versions of a schemaversions=storage.versions("postgresql://localhost:5432/testdb/public/test_table")# List all schemas in the registryschemas=storage.ls()DockerRecap's gateway and registry are also available as a Docker image:dockerrun\-p8000:8000\-eRECAP_URLS=["postgresql://user:pass@localhost:5432/testdb"]'\ghcr.io/recap-build/recap:latestSeeRecap's Docker documentationfor more details.SchemaSeeRecap's type specfor details on Recap's type system.DocumentationRecap's documentation is available atrecap.build.