arm: build: Maintain decoder objects separate from infrastructure objects