Repository: jupyterhub/jupyterhub

Issue:
/hub/metrics total_users double the actual value
As @ianabc reported in https://github.com/jupyterhub/jupyterhub/issues/2685#issuecomment-732310172, there is a discrepancy for the `total_users` metric reported in `/hub/metrics` with the total users reported in `/hub/admin`.

I'm using a JupyterHub 1.2.1. I would guess that total_users is incremented twice per user by mistake.

## /hub/metrics

```
# HELP total_users total number of users
# TYPE total_users gauge
total_users 960.0
```

## /hub/admin

![image](https://user-images.githubusercontent.com/3837114/100183363-074cf200-2edf-11eb-8e6c-888fda72a917.png)


![image](https://user-images.githubusercontent.com/3837114/100183225-ade4c300-2ede-11eb-8245-6274cfe59d8b.png)



Discussion:
I think I can explain why this is happening; `init_users` in app.py [looks at the database during startup](https://github.com/jupyterhub/jupyterhub/blob/959dfb145ac1a9c5e3cf80980c73fa273f462f0b/jupyterhub/app.py#L1774) and initializes `TOTAL_USERS` from that count. Next, _something_ calls `UserListAPIHandler.get()` which is somehow able to increment `TOTAL_USERS` (in some cases - see below) double counting users. For me, and I suspect for @consideRatio, the _something_ is `jupyterhub-idle-culler`. If I disable that, my counts are correct, but I think the actual problem is with `init_users` and `UserDict.__getitem__`.



I was surprised that [UserListAPIHandler.get()](https://github.com/jupyterhub/jupyterhub/blob/202efae6d85582f2cc80eff8546c3162030129b6/jupyterhub/apihandlers/users.py#L52) was able to change state so I dug a bit deeper. That method has a list comprehension which calls `user_model`, and `user_model` does this 
```python
user = self.users[user.id]
```
The underlying [`UserDict.__getitem__`](https://github.com/jupyterhub/jupyterhub/blob/202efae6d85582f2cc80eff8546c3162030129b6/jupyterhub/user.py#L90) calls `self.add` if the key is an integer, and `UserDict.add`, increments the count via
```python
def add(self, orm_user):
    """Add a user to the UserDict"""
    if orm_user.id not in self:
        self[orm_user.id] = self.from_orm(orm_user)
        TOTAL_USERS.inc()
    return self[orm_user.id]
```

At startup, `init_users` sets `TOTAL_USERS` but it doesn't populate the users attribute. Later on, when something calls this method, the `if orm_user.id not in self` condition passes and users get double counted. 

It looks like the `add` method was designed to handle the case that the user is already in `self.users` so I made [this branch](https://github.com/jupyterhub/jupyterhub/compare/master...ianabc:fix-user-init) which alters `init_users` to populate `self.users`. This means the `if orm_user.id not in self` condition fails on subsequent calls and it seems to fix the problem. I'd be (extremely!) happy to open a pull request if that helps, but I wanted to check I was on the right path before going too far. Also, is the `self.add` in `UserDict.__getitem__` OK? It doesn't seem like that method should be allowed to update the users list.



Resolving patch:
diff --git a/jupyterhub/handlers/base.py b/jupyterhub/handlers/base.py
--- a/jupyterhub/handlers/base.py
+++ b/jupyterhub/handlers/base.py
@@ -40,6 +40,7 @@
 from ..metrics import ServerPollStatus
 from ..metrics import ServerSpawnStatus
 from ..metrics import ServerStopStatus
+from ..metrics import TOTAL_USERS
 from ..objects import Server
 from ..spawner import LocalProcessSpawner
 from ..user import User
@@ -453,6 +454,7 @@ def user_from_username(self, username):
             # not found, create and register user
             u = orm.User(name=username)
             self.db.add(u)
+            TOTAL_USERS.inc()
             self.db.commit()
             user = self._user_from_orm(u)
         return user
diff --git a/jupyterhub/user.py b/jupyterhub/user.py
--- a/jupyterhub/user.py
+++ b/jupyterhub/user.py
@@ -69,7 +69,6 @@ def add(self, orm_user):
         """Add a user to the UserDict"""
         if orm_user.id not in self:
             self[orm_user.id] = self.from_orm(orm_user)
-            TOTAL_USERS.inc()
         return self[orm_user.id]
 
     def __contains__(self, key):


Test patch:
diff --git a/jupyterhub/tests/mocking.py b/jupyterhub/tests/mocking.py
--- a/jupyterhub/tests/mocking.py
+++ b/jupyterhub/tests/mocking.py
@@ -41,6 +41,7 @@
 from traitlets import default
 from traitlets import Dict
 
+from .. import metrics
 from .. import orm
 from ..app import JupyterHub
 from ..auth import PAMAuthenticator
@@ -327,6 +328,7 @@ async def initialize(self, argv=None):
             user = orm.User(name='user')
             self.db.add(user)
             self.db.commit()
+            metrics.TOTAL_USERS.inc()
 
     def stop(self):
         super().stop()
diff --git a/jupyterhub/tests/test_metrics.py b/jupyterhub/tests/test_metrics.py
new file mode 100644
--- /dev/null
+++ b/jupyterhub/tests/test_metrics.py
@@ -0,0 +1,34 @@
+import json
+
+from .utils import add_user
+from .utils import api_request
+from jupyterhub import metrics
+from jupyterhub import orm
+
+
+async def test_total_users(app):
+    num_users = app.db.query(orm.User).count()
+    sample = metrics.TOTAL_USERS.collect()[0].samples[0]
+    assert sample.value == num_users
+
+    await api_request(
+        app, "/users", method="post", data=json.dumps({"usernames": ["incrementor"]})
+    )
+
+    sample = metrics.TOTAL_USERS.collect()[0].samples[0]
+    assert sample.value == num_users + 1
+
+    # GET /users used to double-count
+    await api_request(app, "/users")
+
+    # populate the Users cache dict if any are missing:
+    for user in app.db.query(orm.User):
+        _ = app.users[user.id]
+
+    sample = metrics.TOTAL_USERS.collect()[0].samples[0]
+    assert sample.value == num_users + 1
+
+    await api_request(app, "/users/incrementor", method="delete")
+
+    sample = metrics.TOTAL_USERS.collect()[0].samples[0]
+    assert sample.value == num_users
diff --git a/jupyterhub/tests/utils.py b/jupyterhub/tests/utils.py
--- a/jupyterhub/tests/utils.py
+++ b/jupyterhub/tests/utils.py
@@ -4,6 +4,7 @@
 import requests
 from certipy import Certipy
 
+from jupyterhub import metrics
 from jupyterhub import orm
 from jupyterhub.objects import Server
 from jupyterhub.utils import url_path_join as ujoin
@@ -97,6 +98,7 @@ def add_user(db, app=None, **kwargs):
     if orm_user is None:
         orm_user = orm.User(**kwargs)
         db.add(orm_user)
+        metrics.TOTAL_USERS.inc()
     else:
         for attr, value in kwargs.items():
             setattr(orm_user, attr, value)