Change blacklist caching mechanism

This should cut down on memory usage within the cache and when making the blacklist

Change-Id: Iac09f5379402932d10d3a3c38c748885dc9f25d7