internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: wandb/wandb

Issue:
[External dep-related issue] test_get_artifact_obj_by_name fails with a particular combo of py37 and moviepy versions
`standalone_tests/artifact_object_reference_test.py::test_get_artifact_obj_by_name` fails if run on `py3.7` (3.7.9) with a combination of some `moviepy`, `imageio` (and friends) versions. 

@tssweeney was able to track this down to an unclosed file descriptor in https://github.com/Zulko/moviepy/blob/7e3e8bb1b739eb6d1c0784b0cb2594b587b93b39/moviepy/video/io/gif_writers.py#L192.


Discussion:

Resolving patch:
diff --git a/wandb/sdk/data_types/video.py b/wandb/sdk/data_types/video.py
--- a/wandb/sdk/data_types/video.py
+++ b/wandb/sdk/data_types/video.py
@@ -1,7 +1,7 @@
 from io import BytesIO
 import logging
 import os
-from typing import Dict, Optional, Sequence, Type, TYPE_CHECKING, Union
+from typing import Any, Dict, Optional, Sequence, Type, TYPE_CHECKING, Union
 
 from wandb import util
 
@@ -18,6 +18,33 @@
     from ..wandb_run import Run as LocalRun
 
 
+# This helper function is a workaround for the issue discussed here:
+# https://github.com/wandb/client/issues/3472
+#
+# Essentially, the issue is that moviepy's write_gif function fails to close
+# the open write / file descripter returned from `imageio.save`. The following
+# function is a simplified copy of the function in the moviepy source code.
+# See https://github.com/Zulko/moviepy/blob/7e3e8bb1b739eb6d1c0784b0cb2594b587b93b39/moviepy/video/io/gif_writers.py#L428
+#
+# Except, we close the writer!
+def write_gif_with_image_io(
+    clip: Any, filename: str, fps: Optional[int] = None
+) -> None:
+    imageio = util.get_module(
+        "imageio",
+        required='wandb.Video requires imageio when passing raw data. Install with "pip install imageio"',
+    )
+
+    writer = imageio.save(
+        filename, duration=1.0 / clip.fps, quantizer=0, palettesize=256, loop=0
+    )
+
+    for frame in clip.iter_frames(fps=fps, dtype="uint8"):
+        writer.append_data(frame)
+
+    writer.close()
+
+
 class Video(BatchableMedia):
     """Format a video for logging to W&B.
 
@@ -115,7 +142,7 @@ def encode(self) -> None:
         try:  # older versions of moviepy do not support logger argument
             kwargs = {"logger": None}
             if self._format == "gif":
-                clip.write_gif(filename, **kwargs)
+                write_gif_with_image_io(clip, filename)
             else:
                 clip.write_videofile(filename, **kwargs)
         except TypeError:


Test patch:
Repository: wandb/wandb

Issue:
wandb.log() 'data' argument name in header mismatches 'row' argument name in docstring
Very minor issue, but wandb.log() function uses the `data` argument name in the function header but the docstring at line 1416 and 1420 references the `row` argument name.

https://github.com/wandb/client/blob/16658d63aa0f91dd8e98f51d766334af9b404d2b/wandb/sdk/wandb_run.py#L1416

The website documentation would also need an update since it refers to `row` in the Arguments table.  https://docs.wandb.ai/ref/python/log


Discussion:

Resolving patch:
diff --git a/wandb/sdk/wandb_run.py b/wandb/sdk/wandb_run.py
--- a/wandb/sdk/wandb_run.py
+++ b/wandb/sdk/wandb_run.py
@@ -1413,11 +1413,11 @@ def log(
         the data on the client side or you may get degraded performance.
 
         Arguments:
-            row: (dict, optional) A dict of serializable python objects i.e `str`,
+            data: (dict, optional) A dict of serializable python objects i.e `str`,
                 `ints`, `floats`, `Tensors`, `dicts`, or any of the `wandb.data_types`.
             commit: (boolean, optional) Save the metrics dict to the wandb server
                 and increment the step.  If false `wandb.log` just updates the current
-                metrics dict with the row argument and metrics won't be saved until
+                metrics dict with the data argument and metrics won't be saved until
                 `wandb.log` is called with `commit=True`.
             step: (integer, optional) The global step in processing. This persists
                 any non-committed earlier steps but defaults to not committing the


Test patch:
Repository: wandb/wandb

Issue:
Fix mailbox phase 1 locking problem



Discussion:

Resolving patch:
diff --git a/wandb/sdk/lib/mailbox.py b/wandb/sdk/lib/mailbox.py
--- a/wandb/sdk/lib/mailbox.py
+++ b/wandb/sdk/lib/mailbox.py
@@ -32,8 +32,8 @@ def __init__(self, address: str) -> None:
 
     def _get_and_clear(self, timeout: float) -> Optional[pb.Result]:
         found = None
-        with self._lock:
-            if self._event.wait(timeout=timeout):
+        if self._event.wait(timeout=timeout):
+            with self._lock:
                 found = self._result
                 self._event.clear()
         return found


Test patch:
Repository: wandb/wandb

Issue:
[Feature]: Speed up filtering files in a directory with excluded sub-directories
### Description

Currently, when calling `wandb.sdk.lib.filenames.filtered_dir`, the function iterates over every file and folder in the root directory.
The problem starts if there is a directory containing a lot of files that needs to be excluded - in such case, the function will iterate all the files in this directory for no reason.

### Suggested Solution

The function current implementation:
```
def filtered_dir(
    root: str, include_fn: Callable[[str], bool], exclude_fn: Callable[[str], bool]
) -> Generator[str, None, None]:
    """Simple generator to walk a directory"""
    for dirpath, _, files in os.walk(root):
        for fname in files:
            file_path = os.path.join(dirpath, fname)
            if include_fn(file_path) and not exclude_fn(file_path):
                yield file_path
```

Instead of ignoring the directories, one can iterate over the directories as well and exclude them if needed. The change is implemented in the following code snippet:

```
def filtered_dir(
    root: str, include_fn: Callable[[str], bool], exclude_fn: Callable[[str], bool]
) -> Generator[str, None, None]:
    """Simple generator to walk a directory"""
    for dirpath, dirs, files in os.walk(root):
        for dir in dirs:
            if exclude_fn(dir):
                dirs.remove(dir)
        for fname in files:
            file_path = os.path.join(dirpath, fname)
            if include_fn(file_path) and not exclude_fn(file_path):
                yield file_path
```

### Alternatives

_No response_

### Additional Context

_No response_


Discussion:
Hi @amitabe!

Thank you for your suggestion, I can create an internal feature request for this, or if you would like - you can submit a PR to this repository as well.

Thanks,
Ramit

Resolving patch:
diff --git a/wandb/sdk/lib/filenames.py b/wandb/sdk/lib/filenames.py
--- a/wandb/sdk/lib/filenames.py
+++ b/wandb/sdk/lib/filenames.py
@@ -41,6 +41,7 @@ def filtered_dir(
     Yields:
         Generator[str, None, None]: A generator of file paths.
     """
+
     for dirpath, dirs, files in os.walk(root):
         for dir in dirs:
             if exclude_fn(dir):


Test patch:
Repository: wandb/wandb

Issue:
[CLI]: Make class `Runs` sliceable
### Describe the bug

`class Runs(Paginator)` inherits from `Paginator` which defines `__getitem__` in a way that doesn't support slicing, causing the following snippet to throw a `TypeError`:

```python
runs = wandb.Api().runs("<entity>/<project>)

runs[:3]
>>> TypeError: '>' not supported between instances of 'slice' and 'int'

wandb/apis/public.py:1047, in Paginator.__getitem__(self, index)
   1045 def __getitem__(self, index):
   1046     loaded = True
-> 1047     while loaded and index > len(self.objects) - 1:
   1048         loaded = self._load_page()
   1049     return self.objects[index]
```


### Additional Files

_No response_

### Environment

WandB version: 0.13.5

OS: macOS

Python version: 3.11.0

Versions of relevant libraries:


### Additional Context

_No response_


Discussion:

Resolving patch:
diff --git a/wandb/apis/public.py b/wandb/apis/public.py
--- a/wandb/apis/public.py
+++ b/wandb/apis/public.py
@@ -1056,7 +1056,8 @@ def _load_page(self):
 
     def __getitem__(self, index):
         loaded = True
-        while loaded and index > len(self.objects) - 1:
+        stop = index.stop if isinstance(index, slice) else index
+        while loaded and stop > len(self.objects) - 1:
             loaded = self._load_page()
         return self.objects[index]
 


Test patch:
Repository: wandb/wandb

Issue:
[CLI]: Possibly broken type hint for wandb.run
### Describe the bug

My IDE (Visual Studio Code) is unable to infer the type of `wandb.run`, giving it as `Unknown | None`. 

Line 124 of wandb's `__init__.py` is as follows:

```python
run: Optional["wandb.sdk.wandb_run.Run"] = None
```

Elsewhere in the file, `wandb.sdk` is referred to using the variable `wandb_sdk`, which is set on line 26 of `__init__.py`. Modifying line 124 as follows fixes the type hinting for me:

```python
run: Optional["wandb_sdk.wandb_run.Run"] = None
```

I'm not sure if this is a genuine issue with the type hint on line 124, or if VSCode (or Pylance?) should be able to handle this anyway.

### Additional Files

_No response_

### Environment

WandB version: 0.13.6
Pylance version: 2022.11.40
VSCode version: 1.73.1

### Additional Context

_No response_


Discussion:

Resolving patch:
diff --git a/wandb/__init__.py b/wandb/__init__.py
--- a/wandb/__init__.py
+++ b/wandb/__init__.py
@@ -121,7 +121,7 @@ def _assert_is_user_process():
 # globals
 Api = PublicApi
 api = InternalApi()
-run: Optional["wandb.sdk.wandb_run.Run"] = None
+run: Optional["wandb_sdk.wandb_run.Run"] = None
 config = _preinit.PreInitObject("wandb.config", wandb_sdk.wandb_config.Config)
 summary = _preinit.PreInitObject("wandb.summary", wandb_sdk.wandb_summary.Summary)
 log = _preinit.PreInitCallable("wandb.log", wandb_sdk.wandb_run.Run.log)


Test patch:
Repository: wandb/wandb

Issue:
[Feature]: Add Gymnasium support
### Description

[Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the official drop in replacement for Gym. 

When using `monitor_gym=True` in `wandb.init`, wandb is only looking for the non maintaned [Gym](https://github.com/openai/gym). It would be cool to add support of Gymnasium, removing the necessity to have both packages installed in a project.

### Suggested Solution

In the file [wandb/integration/gym/__init__.py](https://github.com/wandb/wandb/blob/main/wandb/integration/gym/__init__.py`) : 
```
vcr = wandb.util.get_module(
        "gym.wrappers.monitoring.video_recorder",
        required="Couldn't import the gym python package, install with `pip install gym`",
    )
```
could also look for `gymnasium.wrappers.monitoring.video_recorder`

### Alternatives

_No response_

### Additional Context

_No response_


Discussion:
That sounds like a great feature request. I can submit it for you :)

Resolving patch:
diff --git a/wandb/integration/gym/__init__.py b/wandb/integration/gym/__init__.py
--- a/wandb/integration/gym/__init__.py
+++ b/wandb/integration/gym/__init__.py
@@ -1,21 +1,52 @@
 import re
+import sys
 from typing import Optional
 
 import wandb
+import wandb.util
+
+if sys.version_info >= (3, 8):
+    from typing import Literal
+else:
+    from typing_extensions import Literal
+
 
 _gym_version_lt_0_26: Optional[bool] = None
+_required_error_msg = (
+    "Couldn't import the gymnasium python package, "
+    "install with `pip install gymnasium`"
+)
+GymLib = Literal["gym", "gymnasium"]
 
 
 def monitor():
+    """Monitor a gym environment.
+
+    Supports both gym and gymnasium.
+    """
+    gym_lib: Optional[GymLib] = None
+
+    # gym is not maintained anymore, gymnasium is the drop-in replacement - prefer it
+    if wandb.util.get_module("gymnasium") is not None:
+        gym_lib = "gymnasium"
+    elif wandb.util.get_module("gym") is not None:
+        gym_lib = "gym"
+
+    if gym_lib is None:
+        raise wandb.Error(_required_error_msg)
+
     vcr = wandb.util.get_module(
-        "gym.wrappers.monitoring.video_recorder",
-        required="Couldn't import the gym python package, install with `pip install gym`",
+        f"{gym_lib}.wrappers.monitoring.video_recorder",
+        required=_required_error_msg,
     )
 
     global _gym_version_lt_0_26
 
     if _gym_version_lt_0_26 is None:
-        import gym  # type: ignore
+        if gym_lib == "gym":
+            import gym
+        else:
+            import gymnasium as gym  # type: ignore
         from pkg_resources import parse_version
 
         if parse_version(gym.__version__) < parse_version("0.26.0"):
@@ -47,7 +78,7 @@ def del_(self):
     recorder.close = close
     wandb.patched["gym"].append(
         [
-            f"gym.wrappers.monitoring.video_recorder.{vcr_recorder_attribute}",
+            f"{gym_lib}.wrappers.monitoring.video_recorder.{vcr_recorder_attribute}",
             "close",
         ]
     )


Test patch:
Repository: wandb/wandb

Issue:
[CLI]: Fix deprecation warning (and slowness)
### Describe the bug

Here's the stacktrace from where the warning is issued. Should be easy to reproduce with any recent setuptools version.
```
  File ".../lib/python3.9/site-packages/wandb/__init__.py", line 26, in <module>
    from wandb import sdk as wandb_sdk
  File ".../lib/python3.9/site-packages/wandb/sdk/__init__.py", line 4, in <module>
    from .artifacts.artifact import Artifact  # noqa: F401
  File ".../lib/python3.9/site-packages/wandb/sdk/artifacts/artifact.py", line 37, in <module>
    from wandb.apis.normalize import normalize_exceptions
  File ".../lib/python3.9/site-packages/wandb/apis/__init__.py", line 43, in <module>
    from .public import Api as PublicApi  # noqa
  File ".../lib/python3.9/site-packages/wandb/apis/public.py", line 3079, in <module>
    class PythonMongoishQueryGenerator:
  File ".../lib/python3.9/site-packages/wandb/apis/public.py", line 3080, in PythonMongoishQueryGenerator
    from pkg_resources import parse_version
  File ".../lib/python3.9/site-packages/pkg_resources/__init__.py", line 118, in <module>
    warnings.warn(
DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
```

pkg_resources is also an infamously slow import, so this will make the wandb imports much less sluggish

### Additional Files

_No response_

### Environment

WandB version: 0.15.5
setuptools version: 68.0.0
Python version: 3.9

### Additional Context

_No response_


Discussion:

Resolving patch:
diff --git a/wandb/apis/public.py b/wandb/apis/public.py
--- a/wandb/apis/public.py
+++ b/wandb/apis/public.py
@@ -16,7 +16,6 @@
 import json
 import logging
 import os
-import platform
 import shutil
 import sys
 import tempfile
@@ -231,7 +230,10 @@ def server_info(self):
         return self._server_info
 
     def version_supported(self, min_version):
-        from pkg_resources import parse_version
+        try:
+            from packaging.version import Version as parse_version  # noqa: N813
+        except ImportError:
+            from pkg_resources import parse_version
 
         return parse_version(min_version) <= parse_version(
             self.server_info["cliVersionInfo"]["max_cli_version"]
@@ -3423,8 +3425,6 @@ def mongo_to_filter(self, filter):
 
 
 class PythonMongoishQueryGenerator:
-    from pkg_resources import parse_version
-
     SPACER = "----------"
     DECIMAL_SPACER = ";;;"
     FRONTEND_NAME_MAPPING = {
@@ -3464,7 +3464,7 @@ class PythonMongoishQueryGenerator:
         ast.Not: "$not",
     }
 
-    if parse_version(platform.python_version()) >= parse_version("3.8"):
+    if sys.version_info >= (3, 8):
         AST_FIELDS = {
             ast.Constant: "value",
             ast.Name: "id",


Test patch:
Repository: wandb/wandb

Issue:
[CLI]: CLI is excessively restrictive in check for `--host`; does not support private/internal DNS hostnames
### Describe the bug

I currently have a wandb server instance deployed internally on a network, inside a VPN service. I have an internal hostname (let's say `myhost`) under which I can access wandb; http://myhost:1234.

Unfortunately, when using the CLI, I can't log in to this instance:

```
$ wandb login --relogin --host=http://myhost:1234
wandb: You can find your API key in your browser here: http://myhost:1234/authorize
wandb: Paste an API key from your profile and hit enter, or press ctrl+c to quit:
wandb: ERROR Host must be a url in the form https://some.address.com, received http://myhost:1234
```

If I remove the check for `normalized_host` in `sdk/lib/apikey.py`, everything works as expected without any further changes. The check simply makes sure that either a `.` or `localhost` is in the hostname before allowing login, which is an excessively restrictive check.

### Additional Files

_No response_

### Environment

WandB version:

OS: macOS 14.1

Python version: 3.11

Versions of relevant libraries: 0.16.2


### Additional Context

_No response_


Discussion:
I can make the PR for this since it's such a small change, just want to see if people are open to the change and if there should still be checks on the hostname or whether that check should be removed completely.

```
if normalized_host != "localhost" and "." not in normalized_host:
    wandb.termerror(
        f"Host must be a url in the form https://some.address.com, received {host}"
    )
    return None
```
Hi @Jamil , thank you for reporting this. You are welcome to create a PR for this, flag it in this thread. Once done, I will flag ask our SDK team to review and they will make a final decision. 

Resolving patch:
diff --git a/wandb/sdk/lib/apikey.py b/wandb/sdk/lib/apikey.py
--- a/wandb/sdk/lib/apikey.py
+++ b/wandb/sdk/lib/apikey.py
@@ -179,11 +179,6 @@ def write_netrc(host: str, entity: str, key: str) -> Optional[bool]:
         return None
     try:
         normalized_host = urlparse(host).netloc.split(":")[0]
-        if normalized_host != "localhost" and "." not in normalized_host:
-            wandb.termerror(
-                f"Host must be a url in the form https://some.address.com, received {host}"
-            )
-            return None
         netrc_path = get_netrc_file_path()
         wandb.termlog(
             f"Appending key for {normalized_host} to your netrc file: {netrc_path}"


Test patch:
diff --git a/tests/pytest_tests/unit_tests/test_lib/test_apikey.py b/tests/pytest_tests/unit_tests/test_lib/test_apikey.py
--- a/tests/pytest_tests/unit_tests/test_lib/test_apikey.py
+++ b/tests/pytest_tests/unit_tests/test_lib/test_apikey.py
@@ -10,8 +10,3 @@ def test_write_netrc():
             "machine localhost\n" "  login vanpelt\n" "  password %s\n" % api_key
         )
 
-
-def test_write_netrc_invalid_host():
-    api_key = "X" * 40
-    res = wandb_lib.apikey.write_netrc("http://foo", "vanpelt", api_key)
-    assert res is None
Repository: wandb/wandb

Issue:
[Q] Contributing guide out of date
Hi there,

I wanted to take a look at contributing and hit a roadblock with the Contributing guide, specifically this section:

https://github.com/wandb/wandb/blob/2829aafc12541eff629b813ed7be896a916b0a64/CONTRIBUTING.md?plain=1#L236-L247

This file no longer exists and removed as part of #7002 which removed the deprecated `pkg_resources` package, specifically commit c3f39af.

Viewing the [last version of the file](https://github.com/wandb/wandb/blob/44c499418e56b2eb14f7ab04cff7317ae159a4c7/tools/setup_dev_environment.py), it seems that all it does is would be to install `pyenv` and the listed versions, which I can replicate easily.

---

Is there a new recommended way to setup environments or (i.e. not sure if tox can do it itself?) or is it just a matter of removing the lines from the CONTRIBUTING.md? 



Discussion:

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -262,7 +262,7 @@ def bump_core_version(session: nox.Session) -> None:
         )
 
 
-@nox.session(python=False, name="proto-go")
+@nox.session(python=False, name="proto-go", tags=["proto"])
 def proto_go(session: nox.Session) -> None:
     """Generate Go bindings for protobufs."""
     _generate_proto_go(session)
@@ -272,7 +272,7 @@ def _generate_proto_go(session: nox.Session) -> None:
     session.run("./core/scripts/generate-proto.sh", external=True)
 
 
-@nox.session(name="proto-python")
+@nox.session(name="proto-python", tags=["proto"])
 @nox.parametrize("pb", [3, 4])
 def proto_python(session: nox.Session, pb: int) -> None:
     """Generate Python bindings for protobufs.


Test patch:
Repository: wandb/wandb

Issue:
[CLI]: Replace deprecated appdirs with platformdirs
### Describe the bug

from https://github.com/ActiveState/appdirs
`Note: This project has been officially deprecated. You may want to check out https://pypi.org/project/platformdirs/ which is a more active fork of appdirs. Thanks to everyone who has used appdirs. Shout out to ActiveState for the time they gave their employees to work on this over the years.`

relevant discussion: https://github.com/ActiveState/appdirs/issues/79

Platformdirs is a maintained fork from appdirs package:
https://github.com/platformdirs/platformdirs

### Additional Files

_No response_

### Environment

WandB version: v0.16.6

OS: Windows11

Python version: 3.9.18

Versions of relevant libraries: appdirs v1.4.3


### Additional Context

_No response_


Discussion:
@SiHaoShen Thanks a lot for updating us!

Resolving patch:
diff --git a/wandb/env.py b/wandb/env.py
--- a/wandb/env.py
+++ b/wandb/env.py
@@ -16,7 +16,7 @@
 from pathlib import Path
 from typing import List, MutableMapping, Optional, Union
 
-import appdirs  # type: ignore
+import platformdirs  # type: ignore
 
 Env = Optional[MutableMapping]
 
@@ -385,7 +385,7 @@ def get_magic(
 
 
 def get_data_dir(env: Optional[Env] = None) -> str:
-    default_dir = appdirs.user_data_dir("wandb")
+    default_dir = platformdirs.user_data_dir("wandb")
     if env is None:
         env = os.environ
     val = env.get(DATA_DIR, default_dir)
@@ -410,7 +410,7 @@ def get_artifact_fetch_file_url_batch_size(env: Optional[Env] = None) -> int:
 
 def get_cache_dir(env: Optional[Env] = None) -> Path:
     env = env or os.environ
-    return Path(env.get(CACHE_DIR, appdirs.user_cache_dir("wandb")))
+    return Path(env.get(CACHE_DIR, platformdirs.user_cache_dir("wandb")))
 
 
 def get_use_v1_artifacts(env: Optional[Env] = None) -> bool:


Test patch:
diff --git a/tests/standalone_tests/executor_tests/requirements.txt b/tests/standalone_tests/executor_tests/requirements.txt
--- a/tests/standalone_tests/executor_tests/requirements.txt
+++ b/tests/standalone_tests/executor_tests/requirements.txt
@@ -11,6 +11,6 @@ protobuf>=3.19.0,!=4.21.0,<5; sys_platform != 'linux'
 PyYAML
 setproctitle
 setuptools
-appdirs>=1.4.3
-dataclasses; python_version < '3.7'
-typing_extensions; python_version < '3.10'
\ No newline at end of file
+platformdirs
+typing_extensions; python_version < '3.10'
+