Repository: spotify/luigi

Issue:
HdfsTarget commands fail when many targets are instantiated
I've recently added an existence check to a large mapreduce task for which some input files may be missing. With a large enough set of inputs, it will fail every time. I've simplified it to the following code:

``` python
from luigi.contrib.hdfs import HdfsTarget

many_targets = [HdfsTarget('/') for _ in range(2000)]
all(target.exists() for target in many_targets)
```

This will break if I use any past 1000 or so. Here the client uses snakebite. For a more direct triggering, we can also do

``` python
from snakebite.client import AutoConfigClient

clients = [AutoConfigClient() for _ in range(10000)]
all(client.test('/', exists=True) for client in clients)
```

In either case, the bug goes away if I use a generator expression rather than a list comprehension. The problem is that when I'm dealing with objects coming out of luigi calls like input_hadoop, it's too late for me to decide between lists and iterators. I can code around this by instantiating all of my HdfsTargets with the same client, but I'm not sure this is safe. It could also be fixed in luigi if we had get_autoconfig_client return the same object each time. Is there any reason this wouldn't work?



Discussion:
Thanks for the report. @daveFNbuck, do you understand what causes this? I don't understand how you got to the conclusion that making `get_autoconfig_client` returning the same objects resolves the issue.

Anyhow I'm pretty sure `get_autoconfig_client` returns different objects each time out of pure convenience of the implementers and that the object is so cheap and there's no need to cache it. Just keep thread/process-safety in mind when refactoring. :)

The bug comes from a system limit of how many file streams can be opened. I tried to dig into the snakebite code to figure out exactly where it happens. I think it's in the thrift communication. The problem is that each client opens a new communication pipe, so if too many are used at the same time you'll hit the limit. If every HdfsTarget has the same client, they don't open multiple pipes. The following code does not trigger the bug:

``` python
from luigi.contrib.hdfs import HdfsTarget, get_autoconfig_client

fs = get_autoconfig_client()
many_targets = [HdfsTarget('/', fs=fs) for _ in range(2000)]
all(target.exists() for target in many_targets)
```

The bug also does not get triggered if there's a chance to garbage collect between exist calls, which usually happens during normal luigi operation. It's only when you hold more HdfsTargets than your system limit on open files in memory at the same time and operate on them all that there's a problem.

I see clearly now. Thanks for investigating this. Naturally, I would happily review and accept a PR fixing this :)

Just wanted to check if there was a good reason not to cache the autoconfig client, maybe threading/multiprocessing issues?

I see, I unfortunately don't know.

I just ran some tests. It seems to be ok to persist the same client object across processes but not threads. Maybe we can make it safe by checking against threading.current_thread() to make sure we create a new one for each thread?

I suppose the reason it works across processes is because you don't share the variables or anything like that in Python, hence it actually creates multiple objects. I might be wrong though. ^^

Yea I think your solution sounds sane.


Resolving patch:
diff --git a/luigi/contrib/hdfs/clients.py b/luigi/contrib/hdfs/clients.py
--- a/luigi/contrib/hdfs/clients.py
+++ b/luigi/contrib/hdfs/clients.py
@@ -19,33 +19,42 @@
 The implementations of the hdfs clients. The hadoop cli client and the
 snakebite client.
 """
-
+import logging
+import threading
 
 from luigi.contrib.hdfs import config as hdfs_config
 from luigi.contrib.hdfs import snakebite_client as hdfs_snakebite_client
 from luigi.contrib.hdfs import webhdfs_client as hdfs_webhdfs_client
 from luigi.contrib.hdfs import hadoopcli_clients as hdfs_hadoopcli_clients
 import luigi.contrib.target
-import logging
 
 logger = logging.getLogger('luigi-interface')
 
+_AUTOCONFIG_CLIENT = threading.local()
+
 
-def get_autoconfig_client():
+def get_autoconfig_client(client_cache=_AUTOCONFIG_CLIENT):
     """
     Creates the client as specified in the `luigi.cfg` configuration.
     """
-    configured_client = hdfs_config.get_configured_hdfs_client()
-    if configured_client == "webhdfs":
-        return hdfs_webhdfs_client.WebHdfsClient()
-    if configured_client == "snakebite":
-        return hdfs_snakebite_client.SnakebiteHdfsClient()
-    if configured_client == "snakebite_with_hadoopcli_fallback":
-        return luigi.contrib.target.CascadingClient([hdfs_snakebite_client.SnakebiteHdfsClient(),
-                                                     hdfs_hadoopcli_clients.create_hadoopcli_client()])
-    if configured_client == "hadoopcli":
-        return hdfs_hadoopcli_clients.create_hadoopcli_client()
-    raise Exception("Unknown hdfs client " + configured_client)
+    try:
+        return client_cache.client
+    except AttributeError:
+        configured_client = hdfs_config.get_configured_hdfs_client()
+        if configured_client == "webhdfs":
+            client_cache.client = hdfs_webhdfs_client.WebHdfsClient()
+        elif configured_client == "snakebite":
+            client_cache.client = hdfs_snakebite_client.SnakebiteHdfsClient()
+        elif configured_client == "snakebite_with_hadoopcli_fallback":
+            client_cache.client = luigi.contrib.target.CascadingClient([
+                hdfs_snakebite_client.SnakebiteHdfsClient(),
+                hdfs_hadoopcli_clients.create_hadoopcli_client(),
+            ])
+        elif configured_client == "hadoopcli":
+            client_cache.client = hdfs_hadoopcli_clients.create_hadoopcli_client()
+        else:
+            raise Exception("Unknown hdfs client " + configured_client)
+        return client_cache.client
 
 
 def _with_ac(method_name):


Test patch:
diff --git a/test/hdfs_client_test.py b/test/hdfs_client_test.py
new file mode 100644
--- /dev/null
+++ b/test/hdfs_client_test.py
@@ -0,0 +1,28 @@
+import itertools
+import threading
+import unittest
+
+from luigi.contrib.hdfs import get_autoconfig_client
+
+
+class HdfsClientTest(unittest.TestCase):
+    def test_get_autoconfig_client_cached(self):
+        original_client = get_autoconfig_client()
+        for _ in range(100):
+            self.assertIs(original_client, get_autoconfig_client())
+
+    def test_threaded_clients_different(self):
+        clients = []
+
+        def add_client():
+            clients.append(get_autoconfig_client())
+
+        # run a bunch of threads to get new clients in them
+        threads = [threading.Thread(target=add_client) for _ in range(10)]
+        for thread in threads:
+            thread.start()
+        for thread in threads:
+            thread.join()
+
+        for client1, client2 in itertools.combinations(clients, 2):
+            self.assertIsNot(client1, client2)