internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: aimhubio/aim

Issue:
Omegaconf not supported by the pytorch_lightning logger
## 🐛 Bug
The config file of `hparam`, in the format of the `OmegaConf` object, cannot be saved by aim's `pytorch_lightning` logger properly. It will be stored as a string instead of a `dict`.
<img src="https://user-images.githubusercontent.com/26243079/150569109-d99ce018-f0ab-449f-af82-c1f37860c397.png" width="700"/>

It would work only if I manually convert the config to a `dict`.
<img src="https://user-images.githubusercontent.com/26243079/150567877-cc741e2e-ff61-4d2f-b788-ecfddaba17b5.png" width="500"/>


### Environment
- Aim: 3.3.5
- Python: 3.9.7
- pip: 21.3.1
- OS:  Linux
- pytorch-lightning: 1.5.8
- omegaconf: 2.1.1




Discussion:

Resolving patch:
diff --git a/aim/storage/treeutils_non_native.py b/aim/storage/treeutils_non_native.py
new file mode 100644
--- /dev/null
+++ b/aim/storage/treeutils_non_native.py
@@ -0,0 +1,33 @@
+from aim.storage.types import AimObject
+
+
+def from_omegaconf_config(obj):
+    try:
+        from omegaconf import OmegaConf
+    except ModuleNotFoundError:
+        return
+
+    if OmegaConf.is_config(obj):
+        return OmegaConf.to_container(obj, resolve=True)
+
+
+def convert_to_native_object(
+    obj: AimObject,
+    *,
+    strict: bool = True,
+):
+    converters = [
+        from_omegaconf_config
+    ]
+    for func in converters:
+        _obj = func(obj)
+        if _obj is not None:
+            obj = _obj
+            break
+    else:
+        if not strict:
+            obj = repr(obj)
+        else:
+            raise TypeError(f'Unhandled non-native value `{obj}` of type `{type(obj)}`.')
+
+    return obj


Test patch:
Repository: aimhubio/aim

Issue:
Pylance reports code is unreachable
## 🐛 Bug

**![image](https://user-images.githubusercontent.com/4717005/165369539-2fa4ea9d-096e-4886-a553-c4171431daf2.png)**

Pylance reports code is unreachable after creating an `Repo` instance. I believe this is due to the incorrect type hints in the `Repo` class.

Pylance sees `read_only` is a `bool`, so the error is always raised and the instantiation is believed to be never successful.


Discussion:

Resolving patch:
diff --git a/aim/sdk/repo.py b/aim/sdk/repo.py
--- a/aim/sdk/repo.py
+++ b/aim/sdk/repo.py
@@ -92,7 +92,7 @@ class Repo:
 
     tracking_queue = _get_tracking_queue()
 
-    def __init__(self, path: str, *, read_only: bool = None, init: bool = False):
+    def __init__(self, path: str, *, read_only: Optional[bool] = None, init: Optional[bool] = False):
         if read_only is not None:
             raise NotImplementedError
 


Test patch:
Repository: aimhubio/aim

Issue:
Pytorch track_gradients_dists errors out if some parameters don't have gradients
## 🐛 Bug

When collecting gradients for each layer weight of a model, the function `get_model_layers` errors out if some model parameters don't have gradients.

### Expected behavior

Ignore weights if grad is None.

### Environment

- Aim Version (e.g., 3.11.1)
- Python version 3.10
- pip version 22.0
- Any OS




Discussion:

Resolving patch:
diff --git a/aim/sdk/adapters/pytorch.py b/aim/sdk/adapters/pytorch.py
--- a/aim/sdk/adapters/pytorch.py
+++ b/aim/sdk/adapters/pytorch.py
@@ -60,15 +60,17 @@ def get_model_layers(model, dt, parent_name=None):
             layers.update(get_model_layers(m, dt, layer_name))
         else:
             layers[layer_name] = {}
-            if hasattr(m, 'weight') \
-                    and m.weight is not None \
-                    and hasattr(m.weight, dt):
-                layers[layer_name]['weight'] = get_pt_tensor(getattr(m.weight, dt)).numpy()
+            weight = None
+            if hasattr(m, 'weight') and m.weight is not None:
+                weight = getattr(m.weight, dt, None)
+                if weight is not None:
+                    layers[layer_name]['weight'] = get_pt_tensor(weight).numpy()
 
-            if hasattr(m, 'bias') \
-                    and m.bias is not None \
-                    and hasattr(m.bias, dt):
-                layers[layer_name]['bias'] = get_pt_tensor(getattr(m.bias, dt)).numpy()
+            bias = None
+            if hasattr(m, 'bias') and m.bias is not None:
+                bias = getattr(m.bias, dt, None)
+                if bias is not None:
+                    layers[layer_name]['bias'] = get_pt_tensor(bias).numpy()
 
     return layers
 


Test patch:
Repository: aimhubio/aim

Issue:
Pytorch Lightning Adapter expects versions to consist of `int`s
The [Pytorch Lightning Adapter](https://github.com/aimhubio/aim/blob/15e394cc39b9def1a283f7e1ca7e6aac7a6d2027/aim/sdk/adapters/pytorch_lightning.py) in 3.14.3 extracts a version tuple from the version string. It expects all parts to map to `int`:

```python
    def versiontuple(v):
        return tuple(map(int, (v.split("."))))
```

I noticed the problem when I received the following exception: `ValueError: invalid literal for int() with base 10: 'post1'`

I suggest using [`packaging.version.parse`](https://packaging.pypa.io/en/latest/version.html#packaging.version.parse), so I'll submit a PR soonish.


Discussion:

Resolving patch:
diff --git a/aim/sdk/adapters/pytorch_lightning.py b/aim/sdk/adapters/pytorch_lightning.py
--- a/aim/sdk/adapters/pytorch_lightning.py
+++ b/aim/sdk/adapters/pytorch_lightning.py
@@ -2,13 +2,12 @@
 from typing import Any, Dict, Optional, Union
 from argparse import Namespace
 
+import packaging.version
+
 try:
     import pytorch_lightning as pl
 
-    def versiontuple(v):
-        return tuple(map(int, (v.split("."))))
-
-    if versiontuple(pl.__version__) < (1, 7):
+    if packaging.version.parse(pl.__version__) < packaging.version.parse("1.7"):
         from pytorch_lightning.loggers.base import (
             LightningLoggerBase as Logger,
             rank_zero_experiment,
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -74,6 +74,7 @@ def package_files(directory):
     # fastapi to support python3.6
     'async-exit-stack>=1.0.0; python_version<"3.7"',
     'async-generator>=1.0; python_version<"3.7"',
+    'packaging>=15.0',
 ]
 
 if platform.machine() != 'arm64':


Test patch:
Repository: aimhubio/aim

Issue:
Detection of Jax Arrays Breaks on Jax=0.4.*
The detection for jax arrays here https://github.com/aimhubio/aim/blob/285a09fefbf25d418e00bc305e9a1357d6343d3c/aim/sdk/num_utils.py#L66 is broken in Jax 0.4.* since `DeviceArray` was renamed to `Array`. The simple fix to support the prior/current version would be to check for either `DeviceArray` or `Array`


Discussion:
Hey @EntilZha. Thanks for raising this issue! Seems this kind of issues can't be avoided without setting upper limit version on third party packages.
@mihran113, could you please take a look?
Long term, likely right @alberttorosyan, but I doubt they'll be renaming core classes frequently :). Setting an upper limit on version would also require adding the dep (possibly large) and then not allow a workaround of:
1) Managing deps with poetry/etc
2) Using aim
3) Using breaking version of jax
4) But, handling the array unwrapping in my trainer code until the fix is in aim.

Thanks!
> Long term, likely right @alberttorosyan, but I doubt they'll be renaming core classes frequently :). Setting an upper limit on version would also require adding the dep (possibly large) and then not allow a workaround of:

Totally agree. Setting upper limit might be too restrictive especially when the package versions do not follow semantic versioning practices.
We recently had similar situation with `sqlalchemy` new release. I guess we can't prevent all of such cases as Python's runtime checking makes it particularly hard to do.
Here is another idea: make it possible for users to register types to unwrap and how to detect them. So I could see something like:
```python

def unwrap_fn(val):
    if isinstance(val, jax.Array):
        return val.item()
    else:
        return val

run = aim.Run(..., unwrap_fn=unwrap_fn)

And this would be plumbed into where things are currently unwrapped. That way, you could include common unwraps, but make it possible to support new versions/obscure unwraps without making training code more verbose (I currently call something like `unwrap_fn` at every call site of `run.track`.
```
`jax.Array` seems like a simplification step from the jax team and a long-term move https://jax.readthedocs.io/en/latest/jax_array_migration.html . It seems like simply adding it to the DeviceArray check should suffice.

Resolving patch:
diff --git a/aim/sdk/num_utils.py b/aim/sdk/num_utils.py
--- a/aim/sdk/num_utils.py
+++ b/aim/sdk/num_utils.py
@@ -67,7 +67,11 @@ def is_jax_device_array(inst):
     """
     Check whether `inst` is instance of jax device array
     """
-    return inst_has_typename(inst, ['jaxlib', 'xla_extension', 'DeviceArray'])
+    if inst_has_typename(inst, ['jaxlib', 'xla_extension', 'Array']):
+        return True
+    if inst_has_typename(inst, ['jaxlib', 'xla_extension', 'DeviceArray']):
+        return True
+    return False
 
 
 def is_numpy_array(inst):


Test patch:
Repository: aimhubio/aim

Issue:
[fix] Serve files linked into the static files directory
In certain conda environments, the files in the static files root directory can be symlinked from a different place. The fixed implementation only resolves relative .. segments in the request path without resolving any symlinks. This way, it still prevents reading arbitrary files through the web server while allowing the reading of symlinked files.


Discussion:
I force-pushed to fix the style issue.
Done!
Hey, I think this PR actually resulted in a 404 for all static files served from `static_files_root`. I guess the condition below should use `!=`. 

https://github.com/aimhubio/aim/blob/main/aim/web/api/views.py#L24-L25
   ```python3
 if common_prefix == static_files_root:
       raise HTTPException(status_code=404)
```

However, note that it would make the code vulnerable to a trivial path traversal: `curl http://127.0.0.1:43800/static-files/x/../../../../../../../../../../../../../secret_gzfile --path-as-is`.

Resolving patch:
diff --git a/aim/web/api/views.py b/aim/web/api/views.py
--- a/aim/web/api/views.py
+++ b/aim/web/api/views.py
@@ -21,7 +21,7 @@ async def serve_static_files(path):
     # Ensure that no paths outside the root directory are accessed by checking that the
     # root directory is a prefix of the file path
     common_prefix = Path(os.path.commonpath([static_files_root, static_file_name]))
-    if common_prefix == static_files_root:
+    if common_prefix != static_files_root:
         raise HTTPException(status_code=404)
 
     compressed_file_name = Path(f'{static_file_name}.gz')


Test patch:
Repository: internetarchive/openlibrary

Issue:
Separate coverstore into separate docker-compose service
Subtask of #1071


Discussion:

Resolving patch:
diff --git a/openlibrary/coverstore/code.py b/openlibrary/coverstore/code.py
--- a/openlibrary/coverstore/code.py
+++ b/openlibrary/coverstore/code.py
@@ -208,11 +208,7 @@ def is_valid_url(url):
             return url.startswith("http://") or url.startswith("https://")
 
         def notfound():
-            if key in ["id", "olid"] and config.get("upstream_base_url"):
-                # this is only used in development
-                base = web.rstrips(config.upstream_base_url, "/")
-                raise web.redirect(base + web.ctx.fullpath)
-            elif config.default_image and i.default.lower() != "false" and not is_valid_url(i.default):
+            if config.default_image and i.default.lower() != "false" and not is_valid_url(i.default):
                 return read_file(config.default_image)
             elif is_valid_url(i.default):
                 raise web.seeother(i.default)


Test patch:
Repository: internetarchive/openlibrary

Issue:
Editing broken on prod
Editing results in Internal Error

```
<type 'exceptions.AttributeError'> at /books/OL4714751M/edit
StringIO instance has no attribute '__exit__'

Python /opt/openlibrary/deploys/openlibrary/a69cf29/openlibrary/plugins/upstream/addbook.py in read_subject, line 645
POST https://openlibrary.org/books/OL4714751M/edit
```

### Evidence / Screenshot (if possible)
![image](https://user-images.githubusercontent.com/6251786/93381255-67b01b00-f82e-11ea-8a55-45ee47bcb896.png)

### Steps to Reproduce
Make any edit

### Details

- **Logged in (Y/N)?** Y
- **Browser type/version?** FF80
- **Operating system?** Win10
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@cclauss



Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/addbook.py b/openlibrary/plugins/upstream/addbook.py
--- a/openlibrary/plugins/upstream/addbook.py
+++ b/openlibrary/plugins/upstream/addbook.py
@@ -642,12 +642,13 @@ def read_subject(subjects):
             if not subjects:
                 return
 
+            f = StringIO(subjects.encode('utf-8'))  # no unicode in csv module
             dedup = set()
-            with StringIO(subjects) as f:
-                for s in next(csv.reader(f, dialect='excel', skipinitialspace=True)):
-                    if s.lower() not in dedup:
-                        yield s
-                        dedup.add(s.lower())
+            for s in next(csv.reader(f, dialect='excel', skipinitialspace=True)):
+                s = s.decode('utf-8')
+                if s.lower() not in dedup:
+                    yield s
+                    dedup.add(s.lower())
 
         work.subjects = list(read_subject(work.get('subjects', '')))
         work.subject_places = list(read_subject(work.get('subject_places', '')))


Test patch:
Repository: internetarchive/openlibrary

Issue:
base64.decodestring was deprecated and removed in Python 3.9 in favor of decodebytes
### Steps to Reproduce

It should be decodestring in python 2 and decodebytes in python 3.

```
openlibrary/plugins/importapi/code.py
462:        username, password = base64.decodestring(authstring).split(':')
606:        username, password = base64.decodestring(authstring).split(':')
```


Discussion:
Hello!
Can I start working on this issue, as it has a good first tag, and I am also new to open-source contributing and would like to fix this issue?
I have pushed a commit, please do review. @tirkarthi 
@tirkarthi I messed up the last pr due to some git issue. I have pushed a fresh pr, do take a look.

Resolving patch:
diff --git a/openlibrary/plugins/importapi/code.py b/openlibrary/plugins/importapi/code.py
--- a/openlibrary/plugins/importapi/code.py
+++ b/openlibrary/plugins/importapi/code.py
@@ -26,7 +26,6 @@
 
 from six.moves import urllib
 
-
 MARC_LENGTH_POS = 5
 logger = logging.getLogger('openlibrary.importapi')
 
@@ -455,11 +454,16 @@ def auth_failed(self, reason):
         d = json.dumps({ "status" : "error", "reason" : reason})
         return web.HTTPError("401 Authorization Required", {"WWW-Authenticate": 'Basic realm="http://openlibrary.org"', "Content-Type": "application/json"}, d)
 
-    def login(self, authstring):
-        if not authstring:
+    def login(self, auth_str):
+        if not auth_str:
             return
-        authstring = authstring.replace("Basic ","")
-        username, password = base64.decodestring(authstring).split(':')
+        auth_str = auth_str.replace("Basic ", "")
+        try:
+            auth_str = base64.decodebytes(bytes(auth_str, 'utf-8'))
+            auth_str = auth_str.decode('utf-8')
+        except AttributeError:
+            auth_str = base64.decodestring(auth_str)
+        username, password = auth_str.split(':')
         accounts.login(username, password)
 
     def prepare_input_data(self, rawdata):
@@ -599,11 +603,16 @@ def build_url(self, url, **params):
         else:
             return url + "?" + urllib.parse.urlencode(params)
 
-    def login(self, authstring):
-        if not authstring:
+    def login(self, auth_str):
+        if not auth_str:
             raise self.auth_failed("No credentials provided")
-        authstring = authstring.replace("Basic ","")
-        username, password = base64.decodestring(authstring).split(':')
+        auth_str = auth_str.replace("Basic ", "")
+        try:
+            auth_str = base64.decodebytes(bytes(auth_str, 'utf-8'))
+            auth_str = auth_str.decode('utf-8')
+        except AttributeError:
+            auth_str = base64.decodestring(auth_str)
+        username, password = auth_str.split(':')
         accounts.login(username, password)
 
     def POST(self):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Library Explorer integrated in openlibrary on dev.openlibrary.org
Subtask of: #3576

Integrate code on https://f1efv.csb.app/ into dev.openlibrary.org


Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/upstream/code.py b/openlibrary/plugins/upstream/code.py
--- a/openlibrary/plugins/upstream/code.py
+++ b/openlibrary/plugins/upstream/code.py
@@ -56,6 +56,13 @@ def GET(self):
         return render_component('HelloWorld') + render_component('HelloWorld')
 
 
+class library_explorer(delegate.page):
+    path = "/explore"
+
+    def GET(self):
+        return render_template('library_explorer')
+
+
 class merge_work(delegate.page):
     path = "(/works/OL\d+W)/merge"
     def GET(self, key):


Test patch:
Repository: internetarchive/openlibrary

Issue:
Koha: fixes failing search case where %3A, i.e. : fails
Addresses a case @JeffKaplan reported from project Koha about certain MARC records which contain ":" colons causing issues.


Discussion:

Resolving patch:
diff --git a/openlibrary/plugins/worksearch/code.py b/openlibrary/plugins/worksearch/code.py
--- a/openlibrary/plugins/worksearch/code.py
+++ b/openlibrary/plugins/worksearch/code.py
@@ -273,14 +273,17 @@ def build_q_list(param):
             if m:
                 q_list.append("author_key:(%s)" % m.group(1))
             else:
-                v = re_to_esc.sub(lambda m:'\\' + m.group(), v)
+                v = re_to_esc.sub(r'\\\g<0>', v)
                 # Somehow v can be empty at this point,
                 #   passing the following with empty strings causes a severe error in SOLR
                 if v:
                     q_list.append("(author_name:(%(name)s) OR author_alternative_name:(%(name)s))" % {'name': v})
 
         check_params = ['title', 'publisher', 'oclc', 'lccn', 'contribtor', 'subject', 'place', 'person', 'time']
-        q_list += ['%s:(%s)' % (k, param[k]) for k in check_params if k in param]
+        q_list += [
+            '%s:(%s)' % (k, re_to_esc.sub(r'\\\g<0>', param[k]))
+            for k in check_params if k in param
+        ]
         if param.get('isbn'):
             q_list.append('isbn:(%s)' % (normalize_isbn(param['isbn']) or param['isbn']))
     return (q_list, use_dismax)
@@ -621,7 +624,7 @@ def GET(self):
             q_list.append(q)
         for k in ('title', 'author', 'isbn', 'subject', 'place', 'person', 'publisher'):
             if k in i:
-                v = re_to_esc.sub(lambda m:'\\' + m.group(), i[k].strip())
+                v = re_to_esc.sub(r'\\\g<0>', i[k].strip())
                 q_list.append(k + ':' + v)
         page = render.work_search(
             i, ' '.join(q_list), do_search, get_doc,


Test patch:
Repository: internetarchive/openlibrary

Issue:
Python 3: could not display: <AttributeError: 'type'>
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->

http://ol-web1.us.archive.org:8080/subjects/biology?debug=true
```
<class 'AttributeError'> at /subjects/biology
can't set attribute
Python	/openlibrary/openlibrary/core/lists/model.py in __init__, line 323
Web	GET http://ol-web1.us.archive.org:8080/subjects/biology
```

### Evidence / Screenshot (if possible)

![Screenshot 2020-11-04 at 20 57 44](https://user-images.githubusercontent.com/3709715/98161763-923e5c00-1ee0-11eb-8fee-85576e75a004.png)

### Relevant url?
<!-- `https://openlibrary.org/...` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Go to ...
2. Do ...

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: 
* Expected: 

### Details

- **Logged in (Y/N)?** 
- **Browser type/version?** 
- **Operating system?** 
- **Environment (prod/dev/local)?** prod
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
https://stackoverflow.com/questions/4183432/cant-set-attribute-with-new-style-properties-in-python

Resolving patch:
diff --git a/openlibrary/core/lists/model.py b/openlibrary/core/lists/model.py
--- a/openlibrary/core/lists/model.py
+++ b/openlibrary/core/lists/model.py
@@ -317,6 +317,7 @@ class Seed:
     """
     def __init__(self, list, value):
         self._list = list
+        self._type = None
 
         self.value = value
         if isinstance(value, six.string_types):
@@ -392,6 +393,8 @@ def _inttime_to_datetime(self, t):
         return datetime.datetime(*time.gmtime(t)[:6])
 
     def get_type(self):
+        if self._type:
+            return self._type
         type = self.document.type.key
 
         if type == "/type/edition":
@@ -405,6 +408,10 @@ def get_type(self):
 
     type = property(get_type)
 
+    @type.setter
+    def type(self, value):
+        self._type = value
+
     def get_title(self):
         if self.type == "work" or self.type == "edition":
             return self.document.title or self.key


Test patch: