A method, comprising: receiving information about a plurality of data chunks, the received information comprising chunk identifying information; broadcasting the chunk identifying information associated with the plurality of data chunks to a plurality of the back-end nodes; one of the plurality of back-end nodes (108a, 108b) designated as a sticky node; receiving bids from each of the back-end nodes, wherein a bid value for each of the plurality of back-end nodes is based on how many data chunks are already stored in the respective back-end node; and comparing the bids to a bid threshold; determining if one or more of a plurality of back-end nodes (108a, 108b) already stores more than the bid threshold amount of the plurality of data chunks, responsive to determining that none of the plurality of back-end nodes (108a, 108b) already stores an amount of the plurality of data chunks that is more than the bid threshold value, deduplicating the plurality of data chunks against the back-end node designated as the sticky node; and responsive to a sticky threshold being reached, designating a different back-end node (108a, 108b) as the next sticky node, wherein the sticky threshold is a threshold on the amount of data being processed since the currently designated sticky node last became the designated sticky node and the sticky threshold varies based on the available storage space of the one or more backend nodes, wherein a least-loaded back-end node is a back-end node having the most available storage space and, if there is a number of least-loaded back-end nodes, a least recently used least-loaded back-end node is designated as the next sticky node. receiving information about a plurality of data chunks, the received information comprising chunk identifying information; broadcasting the chunk identifying information associated with the plurality of data chunks to a plurality of the back-end nodes; one of the plurality of back-end nodes (108a, 108b) designated as a sticky node; receiving bids from each of the back-end nodes, wherein a bid value for each of the plurality of back-end nodes is based on how many data chunks are already stored in the respective back-end node; and comparing the bids to a bid threshold; determining if one or more of a plurality of back-end nodes (108a, 108b) already stores more than the bid threshold amount of the plurality of data chunks, responsive to determining that none of the plurality of back-end nodes (108a, 108b) already stores an amount of the plurality of data chunks that is more than the bid threshold value, deduplicating the plurality of data chunks against the back-end node designated as the sticky node; and responsive to a sticky threshold being reached, designating a different back-end node (108a, 108b) as the next sticky node, wherein the sticky threshold is a threshold on the amount of data being processed since the currently designated sticky node last became the designated sticky node and the sticky threshold varies based on the available storage space of the one or more backend nodes, wherein a least-loaded back-end node is a back-end node having the most available storage space and, if there is a number of least-loaded back-end nodes, a least recently used least-loaded back-end node is designated as the next sticky node.