Repository: DataDog/dd-agent

Issue:
Dogstream fails to monitor log files if they don't exist at startup
If a log file is not preset at startup, dogstreams will fail to monitor the file if it ever appears.

The issue appears to be that `self._gen` in `datadog.py` is never unset on an exception. `TailFile.tail()` raises `StopIteration` if the file files to open, but this is thrown on the first call to `self._gen.next()` instead of when `tail()` is called. Future calls to `self._gen.next()` will just raise `StopIteration`.

I'm not sure if this is by design or not. I'm also not sure if unsetting the generator on an exception is always the right thing to do. However, a file that appears (or re-appears -- e.g. after log rotation) after datadog starts seems like a scenario that should be supported.

If this is not by design. I can prepare a pull request.



Discussion:
Thanks a lot for the great feedback @jalaziz.

This is not something by design. In fact, the Dogstream project is getting quite old, hasn't been updated for a while, and showing a lot of limits.
For this reason, we are currently in the process of looking for a better replacement for it. We'll keep your feedback in mind and let you informed as we get more updates on this.

Also, we'll be happy to review your pull request if you make one.

@jalaziz Thanks for your feedback. Would you be interested in attempting a PR to address this issue? 

@irabinovitch Sure. I'll submit a PR soon.


Resolving patch:
diff --git a/checks/datadog.py b/checks/datadog.py
--- a/checks/datadog.py
+++ b/checks/datadog.py
@@ -186,6 +186,8 @@ def check(self, agentConfig, move_end=True):
             except StopIteration as e:
                 self.logger.exception(e)
                 self.logger.warn("Can't tail %s file" % self.log_path)
+                # reset generator to try again during the next check interval
+                self._gen = None
 
             check_output = self._aggregate(self._values)
             if self._events:


Test patch:
diff --git a/tests/core/test_datadog.py b/tests/core/test_datadog.py
--- a/tests/core/test_datadog.py
+++ b/tests/core/test_datadog.py
@@ -192,6 +192,25 @@ def test_dogstream_bad_input(self):
         actual_output = self.dogstream.check(self.config, move_end=False)
         self.assertEquals(expected_output, actual_output)
 
+    def test_dogstream_io_error(self):
+        log_data = [
+            ('test_metric.e 1000000000 10 metric_type=gauge'),
+        ]
+        expected_output = {"dogstream":
+            [('test_metric.e', 1000000000, 10, self.gauge)]
+        }
+
+        self._write_log(log_data)
+
+        # Simulate missing file by making it unreadable
+        os.chmod(self.log_file.name, 0000)
+        actual_output = self.dogstream.check(self.config, move_end=False)
+        self.assertEquals({}, actual_output)
+        os.chmod(self.log_file.name, 0600)
+        actual_output = self.dogstream.check(self.config, move_end=False)
+        self.assertEquals(expected_output, actual_output)
+
+
     def test_dogstream_log_path_globbing(self):
         """Make sure that globbed dogstream logfile matching works."""
         # Create a tmpfile to serve as a prefix for the other temporary