Fix shuffle buffer being cleared on every shuffle