Do not require {ref, :done}, instead allow sync removal

Signed-off-by: José Valim <jose.valim@plataformatec.com.br>