mesos/spark — pull request #406 by tdas (2013-01-23 07:55:29+00:00)
Title: Changed StorageLevel and BlockManagerId API to present duplication in memory

As StorageLevel/BlockManagerId is deserialized, new objects are created that may have exactly the same value as existing level/id objects. This leads to significant memory bloat specially for streaming as many many blocks are created in this case. Ideally, there should be as many unique id objects as there are workers. Same for storagelevel. This fix ensures that.
For BlockManagerId, the only way to create a new BlockManagerId object is through BlockManagerId.apply(...) which automatically de-dups. I made the constructor private to ensure this. I tried to do the same for StorageLevel. But since StorageLevel is in public API, I kept the main constructor as public so that user programs dont break. However, I have added to the docs saying that the recommended way to create a StorageLevel object is to use StorageLevel.appy(...).