Only keep track of references across "write sessions"