internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: jupyterhub/jupyterhub

Issue:
performance issue about auth
<!-- Thank you for contributing. These HTML commments will not render in the issue, but you can delete them once you've read them if you prefer! -->

### Bug description
There may be performance issue when auth.
in the method of get_current_user_token(), I find some code like 
`db.query(cls).filter(bindparam('prefix', prefix).startswith(cls.prefix))`
corresponding sql as follow 
`select ... FROM api_tokens WHERE ('0ee2' LIKE concat(api_tokens.prefix, '%'))`
explain this sql,
```
select_type: SIMPLE
type:ALL
```
it doesn't use the index api_tokens.ix_api_tokens_prefix.
So when this table has ten thousands of rows, it costs about 1000+ms. it may have performance issue.

#### Expected behaviour
this sql should use the index of api_tokens.ix_api_tokens_prefix.

#### Actual behaviour
Can we change it as follow
`select ... FROM api_tokens WHERE ('0ee2' = api_tokens.prefix)`

### How to reproduce
api_tokens has ten thousands of rows. All requests to jupyterhub always costs N * 1000ms, handred times than beginning.

### Your personal set up
jupyterhub 1.1.0


Discussion:
Hi! Is this still a problem with the latest JupyterHub (2.3.0)? How many users/tokens does it take until you see performance problems?

Resolving patch:
diff --git a/jupyterhub/orm.py b/jupyterhub/orm.py
--- a/jupyterhub/orm.py
+++ b/jupyterhub/orm.py
@@ -528,9 +528,7 @@ def find_prefix(cls, db, token):
         prefix = token[: cls.prefix_length]
         # since we can't filter on hashed values, filter on prefix
         # so we aren't comparing with all tokens
-        prefix_match = db.query(cls).filter(
-            bindparam('prefix', prefix).startswith(cls.prefix)
-        )
+        prefix_match = db.query(cls).filter_by(prefix=prefix)
         prefix_match = prefix_match.filter(
             or_(cls.expires_at == None, cls.expires_at >= cls.now())
         )


Test patch:
Repository: jupyterhub/jupyterhub

Issue:
Add documentation for c.JupyterHub.extra_handlers
This change adds a new `Adding extra handlers to the hub` page to the documentation, to show how to use `c.JupyterHub.extra_handlers` to define custom handlers and have them displayed in the navigation bar:

![image](https://user-images.githubusercontent.com/591645/82819792-4901cf00-9ea1-11ea-8d15-768d62e7ebc1.png)

The main motivation is to make this more visible, as for now `c.JupyterHub.extra_handlers` seems to be mentioned in the api docs and changelog only. 


Discussion:
Additionally there could also be an example to show how to extend from `jupyterhub.apihandlers.APIHandler`.
@jtpio thanks for opening this! Do you think you'll have time anytime soon to address the comments here?
I've marked this as a draft for now

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -1159,7 +1159,7 @@ def _authenticator_default(self):
         Setting this can limit the total resources a user can consume.
 
         If set to 0, no limit is enforced.
-        
+
         Can be an integer or a callable/awaitable based on the handler object:
 
         ::
@@ -1505,14 +1505,21 @@ def add_url_prefix(prefix, handlers):
 
     extra_handlers = List(
         help="""
-        Register extra tornado Handlers for jupyterhub.
-
-        Should be of the form ``("<regex>", Handler)``
+        DEPRECATED.
 
-        The Hub prefix will be added, so `/my-page` will be served at `/hub/my-page`.
+        If you need to register additional HTTP endpoints
+        please use services instead.
         """
     ).tag(config=True)
 
+    @observe("extra_handlers")
+    def _extra_handlers_changed(self, change):
+        if change.new:
+            self.log.warning(
+                "JupyterHub.extra_handlers is deprecated in JupyterHub 3.1."
+                " Please use JupyterHub services to register additional HTTP endpoints."
+            )
+
     default_url = Union(
         [Unicode(), Callable()],
         help="""


Test patch:
Repository: jupyterhub/jupyterhub

Issue:
3.1.1 -> 4.0: upgrade-db just reports "run jupyterhub upgrade-db" again and again
<!-- Thank you for contributing. These HTML comments will not render in the issue, but you can delete them once you've read them if you prefer! -->

### Bug description
After updating from 3.1.1 to 4.0.0 I get (as expected) the message:
```
Found database schema version 651f5419b74d != 0eee8c825d24. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema.
```
So I do that, but the result is just telling the me exact same message again:
```
abel:/opt/jhub # bin/jupyterhub --pid-file /tmp/jhub.pid  --config /etc/jhub/jupyterhub_config.py upgrade-db
Found database schema version 651f5419b74d != 0eee8c825d24. Backup your database and run "jupyterhub upgrade-db" to upgrade to the latest schema.
```

The db_url in  /etc/jhub/jupyterhub_config.py  is this (values masked in <>):
```
abel:/opt/jhub # grep db_url /etc/jhub/jupyterhub_config.py
c.JupyterHub.db_url = 'mysql+mysqlconnector://{}:{}@{}:{}/{}{}'.format("<username>","<pw>","<servername>","<port>","<dbname>","")
```
So I'm a little bit lost... Are there additional steps for upgrading the db from 3.1.1 to 4.0.0? The changelog doesn't mention anything...


### Your personal set up

 - OS: SuSE Linux Enterprise 15 SP4
 - Version(s): jupyterhub 4.0.0, Python 3.10.10, mariadb 10.6.12
 

cu,
Frank




Discussion:

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -9,6 +9,7 @@
 import os
 import re
 import secrets
+import shlex
 import signal
 import socket
 import ssl
@@ -2840,6 +2841,10 @@ async def initialize(self, *args, **kwargs):
         super().initialize(*args, **kwargs)
         if self.generate_config or self.generate_certs or self.subapp:
             return
+        if self.extra_args:
+            self.exit(
+                f"Unrecognized command-line arguments: {' '.join(shlex.quote(arg) for arg in self.extra_args)!r}"
+            )
         self._start_future = asyncio.Future()
 
         def record_start(f):


Test patch:
Repository: jupyterhub/jupyterhub

Issue:
Behavior if server has stopped while hub was down (no Spawner.stop call)
Hi everyone,

JupyterHub checks during the startup if a server is still running. If this is not the case, JupyterHub does not [call](https://github.com/jupyterhub/jupyterhub/blob/3fbdf02cc5c6574432dd28710ed2fc64b2353081/jupyterhub/app.py#L2607) `user.stop(name)` . The server will be set to `None` and removed from the database.  
  
This can cause problems, because `Spawner.post_stop_hook` will never be called in this scenario. One might want to use the configurable `post_stop_hook` as a custom cleanup function, to revert things done previously in `Spawner.pre_spawn_hook`. Or is there a better place to do some cleanup afterwards?  
  
An alternative would be to implement the cleanup in `Spawner.poll` (for `Spawner` subclasses), since this is the only function called during the startup check.

  



Discussion:
I'd need to check the documentation on stop to ensure we require it be able to be called for an already stopped server, but I think this is already the case, and we do call it when start has _failed_. I think calling stop and post-stop hooks in these situations is the right thing to do to be consistent.

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -2785,9 +2785,13 @@ async def check_spawner(user, name, spawner):
                         "%s appears to have stopped while the Hub was down",
                         spawner._log_name,
                     )
-                    # remove server entry from db
-                    db.delete(spawner.orm_spawner.server)
-                    spawner.server = None
+                    try:
+                        await user.stop(name)
+                    except Exception:
+                        self.log.exception(
+                            f"Failed to cleanup {spawner._log_name} which appeared to stop while the Hub was down.",
+                            exc_info=True,
+                        )
                 else:
                     self.log.debug("%s not running", spawner._log_name)
 


Test patch:
Repository: jupyterhub/jupyterhub

Issue:
HTTP 500 error in Docker Jupyterhub when requesting user tokens
Look at this problem ,  it not works with official documents.

https://discourse.jupyter.org/t/http-500-error-in-docker-jupyterhub-when-requesting-user-tokens/18872


Discussion:
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
<br>If you haven't done so already, check out [Jupyter's Code of Conduct](https://github.com/jupyter/governance/blob/master/conduct/code_of_conduct.md). Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
![welcome](https://raw.githubusercontent.com/jupyterhub/.github/master/images/welcome.jpg)
You can meet the other [Jovyans](https://jupyter.readthedocs.io/en/latest/community/content-community.html?highlight=jovyan#what-is-a-jovyan) by joining our [Discourse forum](http://discourse.jupyter.org/). There is also an intro thread there where you can stop by and say Hi! :wave:
<br>Welcome to the Jupyter community! :tada:


Resolving patch:
diff --git a/jupyterhub/apihandlers/users.py b/jupyterhub/apihandlers/users.py
--- a/jupyterhub/apihandlers/users.py
+++ b/jupyterhub/apihandlers/users.py
@@ -472,7 +472,14 @@ async def post(self, user_name):
             user_kind = 'user' if isinstance(user, User) else 'service'
             self.log.info("%s %s requested new API token", user_kind.title(), user.name)
         # retrieve the model
-        token_model = self.token_model(orm.APIToken.find(self.db, api_token))
+        orm_token = orm.APIToken.find(self.db, api_token)
+        if orm_token is None:
+            self.log.error(
+                "Failed to find token after creating it: %r. Maybe it expired already?",
+                body,
+            )
+            raise web.HTTPError(500, "Failed to create token")
+        token_model = self.token_model(orm_token)
         token_model['token'] = api_token
         self.write(json.dumps(token_model))
         self.set_status(201)
diff --git a/jupyterhub/orm.py b/jupyterhub/orm.py
--- a/jupyterhub/orm.py
+++ b/jupyterhub/orm.py
@@ -3,6 +3,7 @@
 # Distributed under the terms of the Modified BSD License.
 import enum
 import json
+import numbers
 from base64 import decodebytes, encodebytes
 from datetime import timedelta
 from functools import partial
@@ -813,7 +814,18 @@ def new(
         else:
             assert service.id is not None
             orm_token.service = service
-        if expires_in is not None:
+        if expires_in:
+            if not isinstance(expires_in, numbers.Real):
+                raise TypeError(
+                    f"expires_in must be a positive integer or null, not {expires_in!r}"
+                )
+            expires_in = int(expires_in)
+            # tokens must always expire in the future
+            if expires_in < 1:
+                raise ValueError(
+                    f"expires_in must be a positive integer or null, not {expires_in!r}"
+                )
+
             orm_token.expires_at = cls.now() + timedelta(seconds=expires_in)
 
         db.commit()


Test patch:
Repository: jupyterhub/jupyterhub

Issue:
`ValueError` thrown for external services in 5.0.0b1 (missing required `oauth_redirect_uri`)
### Bug description

JupyterHub fails to spawn with external services that have `oauth_no_confirm` set:

```
ValueError: Service MyService has oauth configured, but is missing required oauth_redirect_uri.
```

This is probably intentional because it was implemented in https://github.com/jupyterhub/jupyterhub/pull/4555 (which as I understand did not end up being backported in the end). The problem that I have is probably more with documentation for this because the `oauth_redirect_uri` says: _You shouldn't generally need to change this_.

https://github.com/jupyterhub/jupyterhub/blob/8d406c398b1c880bf05465bd287fe49bc9dd8002/jupyterhub/services/service.py#L335-L341

and that it is not documented as one of the inputs, so not really obvious that I could change it:

https://github.com/jupyterhub/jupyterhub/blob/8d406c398b1c880bf05465bd287fe49bc9dd8002/jupyterhub/services/service.py#L161-L174



Discussion:

Resolving patch:
diff --git a/jupyterhub/services/service.py b/jupyterhub/services/service.py
--- a/jupyterhub/services/service.py
+++ b/jupyterhub/services/service.py
@@ -167,6 +167,12 @@ class Service(LoggingConfigurable):
     - url: str (None)
         The URL where the service is/should be.
         If specified, the service will be added to the proxy at /services/:name
+    - oauth_redirect_url: str ('/services/:name/oauth_redirect')
+        The URI for the oauth redirect.
+
+        Not usually needed, but must be set for external services that are not accessed through the proxy,
+        or any service which have a redirect URI different from the default of `/services/:name/oauth_redirect`.
+
     - oauth_no_confirm: bool(False)
         Whether this service should be allowed to complete oauth
         with logged-in users without prompting for confirmation.
@@ -335,7 +341,9 @@ def _validate_client_id(self, proposal):
     oauth_redirect_uri = Unicode(
         help="""OAuth redirect URI for this service.
 
-        You shouldn't generally need to change this.
+        Must be set for external services that are not accessed through the proxy,
+        or any service which have a redirect URI different from the default.
+
         Default: `/services/:name/oauth_callback`
         """
     ).tag(input=True, in_db=False)


Test patch:
Repository: open-mmlab/mmcv

Issue:
Non-portable latest checkpoint symlink
As for now the latest checkpoint created by [runner.py](../blob/master/mmcv/runner/runner.py#L247) is a symlink to a full path of the target checkpoint. Renaming or moving the original folder results into invalid symlink.

Should we change
```python
filename = osp.join(out_dir, filename_tmpl.format(self.epoch + 1))
linkname = osp.join(out_dir, 'latest.pth')
...
mmcv.symlink(filename, linkname)
```
to
```python
filename = osp.join(filename_tmpl.format(self.epoch + 1))
linkname = osp.join(out_dir, 'latest.pth')
...
mmcv.symlink(filename, linkname)
```
to make it relative and thus moving/renaming-proof?

If it's ok I can create a pull request.


Discussion:
It would be nice for a pull request to fix this.

Will this solution work for arbitrary `out_dir`?

Resolving patch:
diff --git a/mmcv/runner/runner.py b/mmcv/runner/runner.py
--- a/mmcv/runner/runner.py
+++ b/mmcv/runner/runner.py
@@ -243,11 +243,13 @@ def save_checkpoint(self,
         else:
             meta.update(epoch=self.epoch + 1, iter=self.iter)
 
-        filename = osp.join(out_dir, filename_tmpl.format(self.epoch + 1))
-        linkname = osp.join(out_dir, 'latest.pth')
+        filename = filename_tmpl.format(self.epoch + 1)
+        filepath = osp.join(out_dir, filename)
+        linkpath = osp.join(out_dir, 'latest.pth')
         optimizer = self.optimizer if save_optimizer else None
-        save_checkpoint(self.model, filename, optimizer=optimizer, meta=meta)
-        mmcv.symlink(filename, linkname)
+        save_checkpoint(self.model, filepath, optimizer=optimizer, meta=meta)
+        # use relative symlink
+        mmcv.symlink(filename, linkpath)
 
     def train(self, data_loader, **kwargs):
         self.model.train()


Test patch:
diff --git a/tests/test_runner.py b/tests/test_runner.py
new file mode 100644
--- /dev/null
+++ b/tests/test_runner.py
@@ -0,0 +1,32 @@
+import os.path as osp
+import tempfile
+import warnings
+
+
+def test_save_checkpoint():
+    try:
+        import torch
+        import torch.nn as nn
+    except ImportError:
+        warnings.warn('Skipping test_save_checkpoint in the absense of torch')
+        return
+
+    import mmcv.runner
+
+    model = nn.Linear(1, 1)
+    runner = mmcv.runner.Runner(
+        model=model,
+        batch_processor=lambda x: x
+    )
+
+    with tempfile.TemporaryDirectory() as root:
+        runner.save_checkpoint(root)
+
+        latest_path = osp.join(root, 'latest.pth')
+        epoch1_path = osp.join(root, 'epoch_1.pth')
+
+        assert osp.exists(latest_path)
+        assert osp.exists(epoch1_path)
+        assert osp.realpath(latest_path) == epoch1_path
+
+        torch.load(latest_path)
Repository: open-mmlab/mmcv

Issue:
mmcv error
My environment is macOS Mojave 10.14.4, Anaconda 4.4.0,Python 3.6.1.
 I directly use "pip install mmcv and got:
"Running setup.py clean for mmcv
Failed to build mmcv
Installing collected packages: mmcv
Running setup.py install for mmcv ... error" and :
"In file included from ./mmcv/video/optflow_warp/flow_warp.cpp:1:
./mmcv/video/optflow_warp/flow_warp.hpp:3:10: fatal error: 'iostream' file not found
#include <iostream>"
Anybody help? Thank you very much.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,6 +1,6 @@
+import platform
 import sys
 from io import open  # for Python 2 (identical to builtin in Python 3)
-
 from setuptools import Extension, find_packages, setup
 
 import numpy
@@ -29,6 +29,13 @@ def get_version():
     return locals()['__version__']
 
 
+if platform.system() == 'Darwin':
+    extra_compile_args = ['-stdlib=libc++']
+    extra_link_args = ['-stdlib=libc++']
+else:
+    extra_compile_args = []
+    extra_link_args = []
+
 EXT_MODULES = [
     Extension(
         name='mmcv._ext',
@@ -37,7 +44,9 @@ def get_version():
             './mmcv/video/optflow_warp/flow_warp_module.pyx'
         ],
         include_dirs=[numpy.get_include()],
-        language="c++",
+        language='c++',
+        extra_compile_args=extra_compile_args,
+        extra_link_args=extra_link_args,
     ),
 ]
 


Test patch:
Repository: open-mmlab/mmcv

Issue:
The function of latest.pth generated by mmcv
What is the function of the latest.pth in mmdetection training generated by mmcv?It only cause trouble when training in colab.I cannot directly set work_dir in google drive path.It will cause os.symlink error.I can only change the source code to solve it. 


Discussion:

Resolving patch:
diff --git a/mmcv/runner/runner.py b/mmcv/runner/runner.py
--- a/mmcv/runner/runner.py
+++ b/mmcv/runner/runner.py
@@ -237,7 +237,8 @@ def save_checkpoint(self,
                         out_dir,
                         filename_tmpl='epoch_{}.pth',
                         save_optimizer=True,
-                        meta=None):
+                        meta=None,
+                        create_symlink=True):
         if meta is None:
             meta = dict(epoch=self.epoch + 1, iter=self.iter)
         else:
@@ -245,11 +246,12 @@ def save_checkpoint(self,
 
         filename = filename_tmpl.format(self.epoch + 1)
         filepath = osp.join(out_dir, filename)
-        linkpath = osp.join(out_dir, 'latest.pth')
         optimizer = self.optimizer if save_optimizer else None
         save_checkpoint(self.model, filepath, optimizer=optimizer, meta=meta)
-        # use relative symlink
-        mmcv.symlink(filename, linkpath)
+        # in some environments, `os.symlink` is not supported, you may need to
+        # set `create_symlink` to False
+        if create_symlink:
+            mmcv.symlink(filename, osp.join(out_dir, 'latest.pth'))
 
     def train(self, data_loader, **kwargs):
         self.model.train()


Test patch:
Repository: open-mmlab/mmcv

Issue:
shape check of impad function
In this [line](https://github.com/open-mmlab/mmcv/blob/8829ff0dfd9b9c9484cbce17f5cecd1128b61723/mmcv/image/transforms/geometry.py#L182):
Instead of `for i in range(len(shape) - 1):
         assert shape[i] >= img.shape[i]`
Should it be
`for i in range(len(shape)):
        assert shape[i] >= img.shape[i]`
to ensure all input shape dimensions equal or less than pad shape?


Discussion:
Thanks for your report! Maybe adding `assert shape[-1] == img.shape[-1]` is better?

Resolving patch:
diff --git a/mmcv/image/transforms/geometry.py b/mmcv/image/transforms/geometry.py
--- a/mmcv/image/transforms/geometry.py
+++ b/mmcv/image/transforms/geometry.py
@@ -180,7 +180,7 @@ def impad(img, shape, pad_val=0):
     if len(shape) < len(img.shape):
         shape = shape + (img.shape[-1], )
     assert len(shape) == len(img.shape)
-    for i in range(len(shape) - 1):
+    for i in range(len(shape)):
         assert shape[i] >= img.shape[i]
     pad = np.empty(shape, dtype=img.dtype)
     pad[...] = pad_val
diff --git a/mmcv/version.py b/mmcv/version.py
--- a/mmcv/version.py
+++ b/mmcv/version.py
@@ -1,2 +1,2 @@
 # Copyright (c) Open-MMLab. All rights reserved.
-__version__ = '0.2.15'
+__version__ = '0.2.16'


Test patch:
diff --git a/tests/test_image.py b/tests/test_image.py
--- a/tests/test_image.py
+++ b/tests/test_image.py
@@ -281,6 +281,16 @@ def test_imcrop(self):
             self.assert_img_equal(patches[i], ref_patch)
 
     def test_impad(self):
+        # grayscale image
+        img = np.random.rand(10, 10).astype(np.float32)
+        padded_img = mmcv.impad(img, (15, 12), 0)
+        assert_array_equal(img, padded_img[:10, :10])
+        assert_array_equal(
+            np.zeros((5, 12), dtype='float32'), padded_img[10:, :])
+        assert_array_equal(
+            np.zeros((15, 2), dtype='float32'), padded_img[:, 10:])
+
+        # RGB image
         img = np.random.rand(10, 10, 3).astype(np.float32)
         padded_img = mmcv.impad(img, (15, 12), 0)
         assert_array_equal(img, padded_img[:10, :10, :])
Repository: open-mmlab/mmcv

Issue:
Bug: Deadlock 'fix' introduced deadlock.
Since https://github.com/open-mmlab/mmcv/pull/252 is merged my mmdetection code hangs after evaluation. After reverting the specific commit `git revert c203419f57c2e25ab4307420b9a3688f99e01dea`, my code runs again as expected..


Discussion:

Resolving patch:
diff --git a/mmcv/runner/hooks/logger/text.py b/mmcv/runner/hooks/logger/text.py
--- a/mmcv/runner/hooks/logger/text.py
+++ b/mmcv/runner/hooks/logger/text.py
@@ -98,16 +98,13 @@ def log(self, runner):
         log_dict['iter'] = runner.inner_iter + 1
         # only record lr of the first param group
         log_dict['lr'] = runner.current_lr()[0]
-        memory = None
-        if torch.cuda.is_available():
-            memory = self._get_max_memory(runner)
         if mode == 'train':
             log_dict['time'] = runner.log_buffer.output['time']
             log_dict['data_time'] = runner.log_buffer.output['data_time']
 
             # statistic memory
-            if memory is not None:
-                log_dict['memory'] = memory
+            if torch.cuda.is_available():
+                log_dict['memory'] = self._get_max_memory(runner)
         for name, val in runner.log_buffer.output.items():
             if name in ['time', 'data_time']:
                 continue


Test patch: