Remove extra asserts, add buffer/source tracking