Repository: DataDog/dd-agent

Issue:
[mongo] "database error: can't use 'local' database through mongos"
System Info:

> Collector (v 5.7.4)
> Platform: Linux-3.10.0-327.18.2.el7.x86_64-x86_64-with-centos-7.2.1511-Core
> Python Version: 2.7.11, 64bit
> MongoDB shell version: 3.2.3
> db.version(): 3.2.3

I'm seeing the following error for certain mongo checks:

```
mongo
-----
  - instance #0 [ERROR]: "database error: can't use 'local' database through mongos"
  - Collected 33 metrics, 0 events & 2 service checks
```

I see there was a pull request merged 29 days ago (#2429) that added code with the comment:

> Fetch information analogous to Mongo's db.getReplicationInfo()`.

It seems that pull request is the culprit here, since it explicitly uses the local database to fetch certain stats:

Line 763:

```
localdb = cli['local']
...
localdb.command("collstats", ol_collection_name)['size'] / 2.0 ** 20, 2
```

Up until now this check has been working fine when used through `mongos`. Is it correct to assume that this check is only meant to be used to directly hit a mongo server, instead of going through `mongos`? Or can this check be updated so it doesn't rely on the `local` database?



Discussion:
Sorry for the late answer @theartoflogic.

Would you mind [reaching out our support team with a flare archive](https://github.com/DataDog/dd-agent/wiki/Send-logs-to-support-using-flare) please ? This would be a valuable feedback to help us tackling this issue.

Yann

Thanks for the follow-up. I'll try sending the flare archive today.

I tried sending a flare archive and I received the following error:

> The upload failed:
> Your request is incorrect: User '<email>' not found

...where `<email>` was my email address.

The full output of the command is as follows:

```
# service datadog-agent flare
2016-05-27 22:37:28,281 | INFO | dd.collector | utils.flare(flare.py:148) | Collecting logs and configuration files:
2016-05-27 22:37:28,281 | INFO | dd.collector | utils.flare(flare.py:382) |   * /var/log/datadog/collector.log.1
2016-05-27 22:37:31,172 | INFO | dd.collector | utils.flare(flare.py:382) |   * /var/log/datadog/collector.log
2016-05-27 22:37:31,763 | INFO | dd.collector | utils.flare(flare.py:382) |   * /var/log/datadog/forwarder.log
2016-05-27 22:37:32,433 | INFO | dd.collector | utils.flare(flare.py:382) |   * /var/log/datadog/dogstatsd.log
2016-05-27 22:37:32,436 | INFO | dd.collector | utils.flare(flare.py:382) |   * /var/log/datadog/jmxfetch.log
2016-05-27 22:37:32,450 | INFO | dd.collector | utils.flare(flare.py:382) |   * /var/log/datadog/supervisord.log
2016-05-27 22:37:32,451 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/datadog.conf - this file contains a credential (api_key) which has been removed in the collected version
2016-05-27 22:37:32,451 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/supervisor.conf
2016-05-27 22:37:32,453 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/docker_daemon.yaml
2016-05-27 22:37:32,453 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/mongo.yaml
2016-05-27 22:37:32,454 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/process.yaml
2016-05-27 22:37:32,455 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/agent_metrics.yaml.default
2016-05-27 22:37:32,455 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/disk.yaml.default
2016-05-27 22:37:32,456 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/network.yaml.default
2016-05-27 22:37:32,456 | INFO | dd.collector | utils.flare(flare.py:393) |   * /etc/dd-agent/conf.d/ntp.yaml.default
2016-05-27 22:37:32,457 | INFO | dd.collector | utils.flare(flare.py:152) |   * datadog-agent configcheck output
2016-05-27 22:37:32,459 | INFO | dd.collector | utils.flare(flare.py:154) |   * datadog-agent status output
2016-05-27 22:37:33,164 | INFO | dd.collector | utils.flare(flare.py:156) |   * datadog-agent info output
2016-05-27 22:37:33,295 | INFO | dd.collector | utils.flare(flare.py:159) |   * pip freeze
2016-05-27 22:37:34,684 | INFO | dd.collector | utils.flare(flare.py:163) |   * log permissions on collected files
2016-05-27 22:37:34,685 | INFO | dd.collector | utils.flare(flare.py:168) | Saving all files to /tmp/datadog-agent-2016-05-27-22-37-28.tar.bz2
/tmp/datadog-agent-2016-05-27-22-37-28.tar.bz2 is going to be uploaded to Datadog.
Do you want to continue [Y/n]?
Please enter your email: <email>
2016-05-27 22:37:49,291 | INFO | dd.collector | utils.flare(flare.py:209) | Uploading /tmp/datadog-agent-2016-05-27-22-37-28.tar.bz2 to Datadog Support
The upload failed:
Your request is incorrect: User '<email>' not found
```

My mistake, I was using the wrong email address. I sent the flare archive. The case ID given is `54398`.


Resolving patch:
diff --git a/checks.d/mongo.py b/checks.d/mongo.py
--- a/checks.d/mongo.py
+++ b/checks.d/mongo.py
@@ -828,44 +828,49 @@ def total_seconds(td):
             except Exception, e:
                 self.log.warning('Failed to record `top` metrics %s' % str(e))
 
-        # Fetch information analogous to Mongo's db.getReplicationInfo()
-        localdb = cli['local']
 
-        oplog_data = {}
+        if 'local' in dbnames: # it might not be if we are connectiing through mongos
+            # Fetch information analogous to Mongo's db.getReplicationInfo()
+            localdb = cli['local']
+
+            oplog_data = {}
+
+            for ol_collection_name in ("oplog.rs", "oplog.$main"):
+                ol_metadata = localdb.system.namespaces.find_one({"name": "local.%s" % ol_collection_name})
+                if ol_metadata:
+                    break
 
-        for ol_collection_name in ("oplog.rs", "oplog.$main"):
-            ol_metadata = localdb.system.namespaces.find_one({"name": "local.%s" % ol_collection_name})
             if ol_metadata:
-                break
+                try:
+                    oplog_data['logSizeMB'] = round(
+                        ol_metadata['options']['size'] / 2.0 ** 20, 2
+                    )
 
-        if ol_metadata:
-            try:
-                oplog_data['logSizeMB'] = round(
-                    ol_metadata['options']['size'] / 2.0 ** 20, 2
-                )
+                    oplog = localdb[ol_collection_name]
 
-                oplog = localdb[ol_collection_name]
+                    oplog_data['usedSizeMB'] = round(
+                        localdb.command("collstats", ol_collection_name)['size'] / 2.0 ** 20, 2
+                    )
 
-                oplog_data['usedSizeMB'] = round(
-                    localdb.command("collstats", ol_collection_name)['size'] / 2.0 ** 20, 2
-                )
+                    op_asc_cursor = oplog.find().sort("$natural", pymongo.ASCENDING).limit(1)
+                    op_dsc_cursor = oplog.find().sort("$natural", pymongo.DESCENDING).limit(1)
+
+                    try:
+                        first_timestamp = op_asc_cursor[0]['ts'].as_datetime()
+                        last_timestamp = op_dsc_cursor[0]['ts'].as_datetime()
+                        oplog_data['timeDiff'] = total_seconds(last_timestamp - first_timestamp)
+                    except (IndexError, KeyError):
+                        # if the oplog collection doesn't have any entries
+                        # if an object in the collection doesn't have a ts value, we ignore it
+                        pass
+                except KeyError:
+                    # encountered an error trying to access options.size for the oplog collection
+                    self.log.warning(u"Failed to record `ReplicationInfo` metrics.")
 
-                op_asc_cursor = oplog.find().sort("$natural", pymongo.ASCENDING).limit(1)
-                op_dsc_cursor = oplog.find().sort("$natural", pymongo.DESCENDING).limit(1)
+            for (m, value) in oplog_data.iteritems():
+                submit_method, metric_name_alias = \
+                    self._resolve_metric('oplog.%s' % m, metrics_to_collect)
+                submit_method(self, metric_name_alias, value, tags=tags)
 
-                try:
-                    first_timestamp = op_asc_cursor[0]['ts'].as_datetime()
-                    last_timestamp = op_dsc_cursor[0]['ts'].as_datetime()
-                    oplog_data['timeDiff'] = total_seconds(last_timestamp - first_timestamp)
-                except (IndexError, KeyError):
-                    # if the oplog collection doesn't have any entries
-                    # if an object in the collection doesn't have a ts value, we ignore it
-                    pass
-            except KeyError:
-                # encountered an error trying to access options.size for the oplog collection
-                self.log.warning(u"Failed to record `ReplicationInfo` metrics.")
-
-        for (m, value) in oplog_data.iteritems():
-            submit_method, metric_name_alias = \
-                self._resolve_metric('oplog.%s' % m, metrics_to_collect)
-            submit_method(self, metric_name_alias, value, tags=tags)
+        else:
+            self.log.debug('"local" database not in dbnames. Not collecting ReplicationInfo metrics')


Test patch: