I think we are confusing two limits here, the normal operation of an spider using this middleware should be like this:

  1) Spider starts gets N urls from HCF (where N is determined by HS_MAX_LINKS)
  2) Spider processes N urls and enters idle_state.
  3) In idle state we do the following:
         3.1) Store the new found URLs in the HCF.
         3.2) Delete the processed batches ids
         3.3) Get N new urls from HCF (this must be done after deleting the previous batches ids or we will get duplicated URLs).
         3.4) Go back to 2.
  4) The spider will continue in this loop until it finishes (either the HCF is empty or we set a limit with the CloseSpider extension)

  HS_MAX_LINKS should be used only to control the number of URLs retrieved from the HCF per iteration, it shouldn't be used to control the whole spider termination.