Massive rewrite for api and ingestion