Repository: voxel51/fiftyone

Issue:
Documentation build takes a long time
Reproducible with `SPHINXOPTS=-E time ./docs/generate_docs.bash` - notably, it takes multiple minutes to build this page:
```
reading sources... [ 18%] api/fiftyone.core.odm                                                 
```

This is at least part of the reason why documentation builds on GitHub actions are taking so long.

Possibly related to #182, although I don't think earlier builds on GitHub actions ran into this.


Discussion:
I got a clean build down to around 20 seconds by spinning up mongodb in a separate fiftyone session. I suspect the issue was introduced in https://github.com/voxel51/fiftyone/commit/20cad388bfaa939bb29ccb904bbbc42527bfc102#diff-a0dd39e588305e0b95c881622442d6f7R15 - even though the DB service isn't starting, it appears that something internal to pymongo is trying to connect to it.

I was able to get this traceback by passing `-vv` in SPHINXOPTS:
```
Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/__init__.py", line 994, in emit
    msg = self.format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 840, in format
    return fmt.format(record)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/util/logging.py", line 490, in format
    message = super().format(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 577, in format
    record.message = record.getMessage()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/util/logging.py", line 99, in getMessage
    message = super().getMessage()
  File "/usr/lib/python3.6/logging/__init__.py", line 338, in getMessage
    msg = msg % self.args
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/mongoengine/queryset/queryset.py", line 73, in __repr__
    self._populate_cache()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/mongoengine/queryset/queryset.py", line 129, in _populate_cache
    self._result_cache.append(next(self))
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1590, in __next__
    raw_doc = next(self._cursor)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/cursor.py", line 1156, in next
    if len(self.__data) or self._refresh():
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/cursor.py", line 1050, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1810, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1763, in __start_session
    server_session = self._get_server_session()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1796, in _get_server_session
    return self._topology.get_server_session()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/topology.py", line 485, in get_server_session
    None)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
Call stack:
  File "/home/alan/.virtualenvs/fiftyone/bin/sphinx-build", line 8, in <module>
    sys.exit(main())
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/cmd/build.py", line 288, in main
    return make_main(argv)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/cmd/build.py", line 195, in make_main
    return make_mode.run_make_mode(argv[1:])
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/cmd/make_mode.py", line 167, in run_make_mode
    return make.run_generic_build(args[0])
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/cmd/make_mode.py", line 155, in run_generic_build
    return build_main(args + opts)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/application.py", line 349, in build
    self.builder.build_update()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 418, in read
    self._read_serial(docnames)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/io.py", line 316, in read_doc
    pub.publish()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/io.py", line 130, in read
    self.parse()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/parsers.py", line 93, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2771, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2771, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/statemachine.py", line 239, in run
    context, state, transitions)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2344, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2356, in explicit_construct
    return method(self, expmatch)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2099, in directive
    directive_class, match, type_name, option_presets)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2148, in run_directive
    result = directive_instance.run()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/directive.py", line 146, in run
    documenter.generate(more_content=self.content)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 734, in generate
    self.document_members(all_members)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 656, in document_members
    check_module=members_check_module and not isattr)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 1199, in generate
    all_members=all_members)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 734, in generate
    self.document_members(all_members)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 1188, in document_members
    super().document_members(all_members)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 656, in document_members
    check_module=members_check_module and not isattr)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 682, in generate
    if not self.import_object():
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 1406, in import_object
    ret = super().import_object()
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py", line 324, in import_object
    warningiserror=self.env.config.autodoc_warningiserror)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/ext/autodoc/importer.py", line 72, in import_object
    logger.debug('[autodoc] => %r', obj)
  File "/usr/lib/python3.6/logging/__init__.py", line 1630, in debug
    self.log(DEBUG, msg, *args, **kwargs)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/util/logging.py", line 124, in log
    super().log(level, msg, *args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1674, in log
    self.logger.log(level, msg, *args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1374, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1444, in _log
    self.handle(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 1454, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 1516, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.6/logging/__init__.py", line 865, in handle
    self.emit(record)
  File "/home/alan/.virtualenvs/fiftyone/lib/python3.6/site-packages/sphinx/util/logging.py", line 165, in emit
    super().emit(record)
```
Commenting out https://github.com/voxel51/fiftyone/blob/9d40af0db611c194f448194dfa680f1a12564cad/fiftyone/core/odm/database.py#L26 takes the time down to around 30s (possibly still some more room for improvement, although this is with a sample size of 1). So this is similar to #182, although I suspect lazily instantiating the DB instance here is easier.

Resolving patch:
diff --git a/fiftyone/core/odm/database.py b/fiftyone/core/odm/database.py
--- a/fiftyone/core/odm/database.py
+++ b/fiftyone/core/odm/database.py
@@ -22,17 +22,23 @@
 import pymongo
 
 _DEFAULT_DATABASE = "fiftyone"
+_client = None
 
 
-_db = connect(_DEFAULT_DATABASE)
+def _connect():
+    global _client
+    if _client is None:
+        connect(_DEFAULT_DATABASE)
+        _client = pymongo.MongoClient()
 
 
 def get_db_conn():
     """Creates a connection to the database"""
-    return pymongo.MongoClient()[_DEFAULT_DATABASE]
+    _connect()
+    return _client[_DEFAULT_DATABASE]
 
 
 def drop_database():
     """Drops the database."""
-    client = pymongo.MongoClient()
-    client.drop_database(_DEFAULT_DATABASE)
+    _connect()
+    _client.drop_database(_DEFAULT_DATABASE)
diff --git a/fiftyone/core/service.py b/fiftyone/core/service.py
--- a/fiftyone/core/service.py
+++ b/fiftyone/core/service.py
@@ -118,6 +118,11 @@ def start(self):
 
         super().start()
 
+        # Set up a default connection
+        import fiftyone.core.odm as foo
+
+        foo.get_db_conn()
+
         # Drop non-persistent datasets
         import fiftyone.core.dataset as fod
 


Test patch:
diff --git a/tests/benchmarking/cifar10_benchmark.log b/tests/benchmarking/cifar10_benchmark.log
--- a/tests/benchmarking/cifar10_benchmark.log
+++ b/tests/benchmarking/cifar10_benchmark.log
@@ -1,3 +1,4 @@
 githash load_dataset read_samples update_modify_field update_new_field delete_samples
 7b4d8b1fe02eb4c9bed7e1b927e9738bc9c6bb0c 39.8293  0.2410  1.3153  1.3597  0.3115
-3eb0ef354e65765a679f271ffea92ddb8bdf13ef 31.0185  0.2451  1.3416  1.6098  0.3663
\ No newline at end of file
+3eb0ef354e65765a679f271ffea92ddb8bdf13ef 31.0185  0.2451  1.3416  1.6098  0.3663
+54dbb2c9ea0120915add4ba1ed77ce386e0f9ea2 24.5463  0.2408  1.3390  1.5883  0.3428
\ No newline at end of file