Actually give the block back if the join/swap was free