mongodb/mongo-hadoop — pull request #117 by adq (2014-10-15 09:20:17+00:00)
Title: Cache MongoClient instances for connection sharing.

Hi, spotted another issue: when I run this code in the interactive spark shell, it leaves all the Mongo connections open. If you run more and more queries, eventually the mongod runs out of connections. Secondly, I only have 8 multithreaded cores running, so I would expect around that number of connections. However, I end up with one connection for each Spark collection "split", which is well over 100 with this limited dataset.

The problem is because the code currently creates a new MongoClient() instance for each connection. It appears you're meant to create a single MongoClient per JVM/db cluster, and use its built in connection pooling. I've added a static MongoClient cache so this happens from now on. I only get about 10 connections to mongo now, and they are reused. This will be much more efficient as it isn't establishing a connection every time (esp with auth enabled).

See: http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-java-driver/
"Typically you create only 1 instance for a given database cluster and use it across your application"