How to deal with cleaning meta-data correctly is a complicated issue and one which is not yet completely addressed in the codebase. The reason this is necessary is streaming jobs which might create thousands of RDD's over a short period of time, so any state really needs to be garbage collected periodically. Otherwise hashmaps like this bloat to the point where they start to impact performance.

For now, TD has introduced these helper data structures which use a timeout based approach. He's added them throughout the codebase, but they are enabled only in streaming mode. In normal batch mode, it acts just like any normal concurrent hashmap. 

Figuring out the correct GC behavior for all types of state is something TD is working on right now, but in the meantime this timeout approach is being employed.